Understanding Key Terms in ML

Core Machine Learning Concepts

Machine Learning (ML)

A subset of Artificial Intelligence (AI) that enables systems to learn from data, identify patterns, and make decisions or predictions with minimal human intervention, without being explicitly programmed for every task.

Features

The independent variables or attributes (columns) in a dataset that are used as input to a machine learning model. These are the characteristics or properties of the data points.

Labels (Target Variable)

The dependent variable or outcome (the column) that a machine learning model is trained to predict. In supervised learning, this is the "answer" the model learns from.

Dataset

A collection of data, typically organized in a tabular format (rows and columns), used for training and evaluating machine learning models.

Training Data

The subset of the dataset used to train a machine learning model. The model learns patterns and relationships from this data.

Test Data

A separate subset of the dataset, held back during training, used to evaluate the model's performance on unseen data. It assesses how well the model generalizes.

Validation Data

An optional third subset of data, used during the model development process to tune hyperparameters and prevent overfitting on the test set. It's distinct from both training and test data.

Model

The output of a machine learning algorithm after it has been trained on data. It represents the learned patterns and relationships, ready to make predictions on new inputs.

Prediction

The output generated by a trained machine learning model when given new, unseen input features.

Generalization

A model's ability to perform accurately on new, unseen data, beyond the specific examples it was trained on. This is a primary goal of machine learning.

Overfitting

Occurs when a model learns the training data too well, including its noise and random fluctuations, leading to excellent performance on training data but poor performance on new, unseen data.

Underfitting

Occurs when a model is too simple to capture the underlying patterns in the training data, resulting in poor performance on both training and test data.

Bias-Variance Trade-off

A central concept in ML. High bias leads to underfitting (model is too simple). High variance leads to overfitting (model is too complex). The goal is to find a balance between the two to optimize generalization.

Types of Machine Learning

Supervised Learning

Learning from labeled data (features and their corresponding correct labels/outcomes). The model learns a mapping from inputs to outputs based on examples. Examples: Classification, Regression.

Unsupervised Learning

Learning from unlabeled data, aiming to find hidden patterns, structures, or relationships within the data without explicit guidance. Examples: Clustering, Dimensionality Reduction.

Reinforcement Learning

An agent learns to make decisions by interacting with an environment, receiving rewards for desired actions and penalties for undesirable ones, to maximize cumulative reward over time.

Data Preparation & Preprocessing

Feature Engineering

The process of creating new features or transforming existing ones from raw data to improve model performance and capture more relevant information.

Feature Selection

The process of selecting a subset of relevant features from the original dataset to use in model training, aiming to reduce dimensionality, improve performance, and reduce overfitting.

Feature Scaling

Techniques (like Standardization or Normalization) used to transform features so they have similar scales. This is crucial for distance-based algorithms (e.g., KNN, SVM) and gradient-based optimizers.

Categorical Encoding

Converting categorical (non-numerical) features into numerical representations that machine learning models can understand. Common methods include One-Hot Encoding and Label Encoding.

Missing Data Handling

Strategies to address missing values in a dataset, which can include imputation (filling in missing values with estimates like mean, median, mode) or removal of rows/columns with missing data.

Supervised Learning Algorithms

Linear Regression

A regression algorithm that models the linear relationship between a dependent variable and one or more independent variables by finding the best-fitting straight line (or hyperplane) through the data.

Key terms: Slope, Intercept, Residuals, Cost Function (MSE).

Logistic Regression

A classification algorithm that models the probability of a binary outcome by transforming the output of a linear equation using a sigmoid function. Used for predicting categories.

Key terms: Sigmoid Function, Probability, Decision Threshold, Cross-Entropy Loss.

Decision Tree

A non-parametric supervised learning algorithm used for both classification and regression. It builds a model in the form of a tree structure, where each internal node represents a test on a feature, each branch represents an outcome of the test, and each leaf node holds a class label or a regression value.

Key terms: Node, Branch, Leaf, Root Node, Split, Impurity (Gini, Entropy), Recursive Partitioning.

K-Nearest Neighbors (KNN)

A non-parametric, instance-based, lazy learning algorithm used for both classification and regression. It classifies a new data point based on the majority class (or average value) of its 'K' nearest neighbors in the training data.

Key terms: K, Nearest Neighbors, Distance Metric (Euclidean Distance).

Support Vector Machine (SVM)

A powerful supervised learning algorithm used primarily for classification. It finds the optimal hyperplane that best separates data points of different classes by maximizing the margin between them.

Key terms: Hyperplane, Margin, Support Vectors, Kernel Trick (Linear, RBF, Polynomial).

Unsupervised Learning Algorithms

K-Means Clustering

An unsupervised clustering algorithm that partitions 'N' data points into 'K' clusters, where each data point belongs to the cluster with the nearest mean (centroid). It iteratively refines cluster assignments and centroid positions.

Key terms: Cluster, Centroid, Inertia, Local Optima, Elbow Method.

Principal Component Analysis (PCA)

A linear dimensionality reduction technique that transforms data into a new set of orthogonal (uncorrelated) variables called Principal Components, capturing most of the data's variance with fewer dimensions.

Key terms: Dimensionality Reduction, Principal Component, Variance, Eigenvector, Eigenvalue, Information Loss.

Ensemble Methods

Ensemble Learning

A technique that combines predictions from multiple individual models (often called "weak learners" or "base learners") to produce a more accurate and robust overall prediction than any single model could achieve.

Bagging (Bootstrap Aggregating)

An ensemble technique where multiple models are trained independently on different random subsets (bootstrap samples) of the training data, and their predictions are then averaged (regression) or voted (classification).

Random Forest

An ensemble method using bagging, specifically with decision trees. It builds a "forest" of many decision trees, each trained on a bootstrap sample and considering only a random subset of features at each split point to ensure diversity.

Boosting

An ensemble technique where models are built sequentially, with each new model focusing on correcting the errors (residuals) made by the previous models in the ensemble. It iteratively combines weak learners to create a strong predictor.

Gradient Boosting Machines (GBM)

A popular boosting algorithm that builds new models (typically decision trees) to predict the residuals (errors) of the previous models, adding their predictions to the ensemble with a "learning rate" to prevent overfitting.

Weak Learner (Base Learner)

A model that performs slightly better than random chance. In ensemble methods like boosting, many weak learners are combined to form a strong learner.

Residuals

The differences between the actual target values and the values predicted by a model. In boosting, subsequent models learn to predict these errors.

Learning Rate (Shrinkage)

A hyperparameter in boosting algorithms that controls the contribution of each weak learner to the overall model. A smaller learning rate requires more trees but often leads to better generalization.

Evaluation Metrics & Validation

Accuracy (Classification)

The proportion of correctly classified instances out of the total instances. Useful when classes are balanced.

$$\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$$

Precision (Classification)

The proportion of true positive predictions among all positive predictions made by the model. Important when the cost of False Positives is high.

$$\text{Precision} = \frac{\text{True Positives}}{\text{True Positives + False Positives}}$$

Recall (Sensitivity, Classification)

The proportion of true positive predictions among all actual positive instances. Important when the cost of False Negatives is high.

$$\text{Recall} = \frac{\text{True Positives}}{\text{True Positives + False Negatives}}$$

F1-Score (Classification)

The harmonic mean of Precision and Recall. Provides a single score that balances both metrics, especially useful with imbalanced classes.

$$\text{F1-Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Confusion Matrix (Classification)

A table that visualizes the performance of a classification model, showing the counts of true positives, true negatives, false positives, and false negatives.

Mean Squared Error (MSE, Regression)

The average of the squared differences between predicted and actual values. Penalizes larger errors more heavily.

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Mean Absolute Error (MAE, Regression)

The average of the absolute differences between predicted and actual values. Less sensitive to outliers than MSE.

$$\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

R-squared ($R^2$, Regression)

Also known as the Coefficient of Determination, it represents the proportion of variance in the dependent variable that can be predicted from the independent variables. Ranges from 0 to 1, with higher values indicating a better fit.

$$R^2 = 1 - \frac{\text{Sum of Squared Residuals (SSR)}}{\text{Total Sum of Squares (SST)}}$$

Cross-Validation

A technique for evaluating model performance by partitioning the data into multiple subsets, training on some, and testing on others, usually multiple times. This provides a more robust estimate of performance and reduces bias from a single train/test split.

Key terms: K-Fold Cross-Validation.

Hyperparameters

Parameters that are set *before* the training process begins, rather than being learned from the data. They control the learning process and model structure (e.g., `K` in KNN, `max_depth` in Decision Trees, `learning_rate` in GBM).

Ask Gemini about ML Terms ✨

Got a question about any Machine Learning term? Type it below and Gemini will help explain it in a concise, easy-to-understand way!

Your answer will appear here...

Thinking...