Mathematics for Machine Learning

Geometrical Interpretation Of Dot Product Of Two Vectors:

Suppose there are two vectors, P and Q.

$P = a_1i + a_2j + a_3k$, and $Q = b_1i + b_2j + b_3k$

where $i,j ,k$ are unit vectors along x, y and z coordinate respectively. Then, P.Q will define the scalar product.

$P.Q = (a_1i + a_2j + a_3k).(b_1i + b_2j + b_3k)$ $P.Q = (a1b1)*(i.i) + (a1b2)*(i.j) + (a1b3)*(i.k) + (a_2b_1)*(j.i) + (a_2b_2)*(j.j)$ $+(a_2b_3)*(j.k)+ (a_3b_1)*(k.i) + (a_3b_2)*(k.j) + (a_3b_3)*(k.k)$ $P.Q = a_1b_1 + a_2b_2 + a_3b_3$

So, If we place vector a on vector b from the point of their intersection, then the length of vector b occupied by vector a is the projection of vector a on vector b. It is like the shadow of vector a falling onto vector b.

The projection of vector a on the vector b is calculated by, $(a.b)/|b|$

The projection of a vector b on vector a is calculated by, $(a.b)/|a|$.

The angle between any two vectors formed by their intersection at one point is calculated by $Cosθ = (a.b)/|a|*|b|$.

Geometrical Interpretation Of Cross Product Of Two Vectors:

Suppose there are two vectors, P and Q.

$P = a_1i + a_2j + a_3k$, and $Q = b_1i + b_2j + b_3k$

Then, P x Q will define the vector product.

$P \times Q = (a_1i + a_2j + a_3k)\times (b_1i + b_2j + b_3k)$

$P \times Q = ((a_2b_3) - (b_2a_3))i - ((a_1b_3) - (b_1a_3))j + ((a_1b_2) - (b_1a_2))k$

If we place vector a on vector b from the point of their intersection, then the length of vector b occupied by vector a is the projection of vector a on vector b. So it is like the shadow of vector a falling onto vector b`.

Matrix and Matrix Operations:

A matrix consists of numbers arranged in rows and columns enclosed in brackets. The dimensions or order of a matrix gives the number of rows followed by the number of columns in a matrix.

The most common form for representing a machine learning dataset is the matrix, where rows correspond to a sample and columns correspond to different attributes or features of a sample.

Matrix Addition:
The addition of two matrices and the scalar multiplication of a matrix with a scalar is similar to vector addition and scalar multiplication of vector, respectively. One can perform matrix addition if the size of two matrices is the same.

Example: Let's consider two matrices, A and B, as follows:

$\begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix}$

$\begin{bmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{bmatrix}$

The vector addition generates $C=A+B=$ $\begin{bmatrix} a_{11}+b_{11} & a_{12}+b_{12} \\ a_{21}+b_{21} & a_{22}+b_{22} \end{bmatrix}$

Matrix Multiplication:
Let us consider two matrices, A and B, having dimensions $a \times b$ and $b \times c$, respectively. The matrix C, the product of matrix A and matrix B, can be written as $C = AB$ and is an $a × c$ dimentionalmatrix. An element in product matrix C, $C_{xy}$ can be defined as $A_{x1}B_{y1} +….. + A_{xb}B_{by} = ∑_{k=1}^b A_{kx}B_{ky}$ for the values $x = 1 ... a$ and $y= 1 ...c$

Multivariate Calculus

Limits:
A limit is a value that a function approaches as its input value approaches some number. Limits are denoted as follows: $\lim_{x \to a} f(x)$ It is read as "the limit of f(x) as x approaches a is equal to L."

Example:
Consider the following function $(x^2-1)/(x-1)$. For x=1, it takes a value of $0/0$, which is undefined. Let's look at the figure below:

As the value of x goes closer to 1, the value of the function $(x^2-1)/(x-1)$ goes closer to 2. So, we can say, "ignoring what happens when $x=1$, as x goes closer to 1, the answer gets closer to 2".

Derivatives:
Given a function $f(x)$ as y, the derivative of f is the rate of change of the $f(x)$ relative to x. The derivative of $f(x)$, denoted $f'(x)$, or $df(x)/dx$, is defined by the following limit: $f'(x)=\Delta y/\Delta x=lim_{h \to 0} (f(x+h)-f(x))/h$.

In fig above, let two points on the curve $f(x)$ be $(x, f(x))$ and $((x + h)$, $f(x + h))$. Here, the slope of the second line through these two points is given by $[f(x + h) - f(x)]/(x + h - x) = [f(x + h) - f(x)] / h$ . If we observe closely when the distance between two points tends to 0 (i.e., as h approaches 0), the second point overlaps the original point, and the secant line becomes the tangent line. The tangent line's slope is called the derivative of the function.

Gradients:
For a function that has more than one input variable, its derivative is called a gradient. Unlike a derivative, a gradient has two components, magnitude, and direction. For a 2D plot, the gradient generally refers to a line's slope.

Example:
Q. Find the gradient of f(x,y) = $3x^2 + 2y + 1$ at $(2,1)$.
A. df(x,y)/dx = 6x, df(x,y)/dy=2. Hence the gradient $(6x,2)$ at $(2,1)$ is $(6*2,2)=(12,2)$

Probability Theory

Events, Probabilities, and Random Variables

Events:
In probability theory, an event can be defined as a set of outcomes of a random experiment. The sample space indicates all possible results of a random experiment. Thus, events in probability are nothing but a subset of the sample space. Let's try to understand with an example: Suppose you roll a fair die. Here, The total number of possible outcomes = $the sample space = {1, 2, 3, 4, 5, 6}$. Let an event, E = getting an even number on the die. Then $E = {2, 4, 6}$. Thus, E is a subset of the sample space and is an outcome of the rolling of a die.

Probability:
Probability is predicting an event based on the study of previous records or the number and type of possible outcomes. It explains the possibility of a particular event occurring. In the example shown above, the probability of event E, i.e., getting an even number on the die, is $3/6=0.5$. A random variable is a type of variable whose value depends upon the numerical outcomes of a specific random phenomenon. Random variables are always real numbers as they are measurable.

Random Variable:
A random variable is a type of variable whose value depends upon the numerical outcomes of a specific random phenomenon. Random variables are always real numbers (either discrete or continuous) as they are measurable. Suppose two dies are rolled, and the random variable, X, represents the sum of the numbers. Then, X can take any integer value between 2 to 12.

Probability Distributions:

Probability Mass Function:

A Probability Mass Function(PMF) describes the probability that a discrete random variable takes on a particular value.
, For example,, suppose we roll a die once. Let x denote the outcome of the die roll, and then the probability for different values of X is as follows:

$P(X=1): 1/6$ $P(X=2): 1/6$ $P(X=3): 1/6$ $P(X=4): 1/6$ $P(X=5): 1/6$ $P(X=6): 1/6$

There is an equal chance for all six numbers. These probabilities as a probability mass function can be written as:

$p_X(x) = \begin{cases} 1/6, \text{ if } x=1\\ 1/6, \text{ if } x=2 \\ 1/6, \text{ if } x=3 \\ 1/6, \text{ if } x=4 \\ 1/6, \text{ if } x=5\\ 1/6, \text{ if } x=6 \end{cases}$

Probability Density Function:

The Probability Density Function(PDF) measures the density of the probability that a continuous random variable will lie within a particular range of values. The probability taken by continuous random variable X on some given value x is always 0; hence probability mass function is not helpful for a continuous random variable. We integrate the probability density function between two specified points to determine this probability.

The figure above depicts the probability that continuous random X will lie between two points, a and b. It can be calculated as P($a \le X \le b$) = $\int_{a}^{b}$f(x)dx

Cumulative Distribution Function:

The Cumulative Distribution Function (CDF) is the probability that the random variable X is less than or equal to a certain number x.
The formula for CDF is:
$F_X(x)=P(X\le x)$

Example:

Q. en PDF below, find $P(X=2)$ and $P(X > 4)$ \begin{matrix} x&1& 2& 3 & 4 \ f(x)& 0.13 & 0.17 & 0.5 &0.2 \end{matrix}
A: P(X=2) = f(X=1) + f(X=2) = 0.13+0.17 = 0.3 P(X > 4)=1-P(X $\le 3$) =1-0.8 = 0.2

Conditional Probability:

Conditional and Marginal Probabilities
Condition probability helps to calculate the probability of one event occurring, given that another event has occurred. Mathematically, it can be written as $P(A/B)=P(A, B)/P(B)$. $P(A, B)$ is called the joint probability distribution.

Let's look at the probability distribution table below.

Suppose we are interested in calculating the probability that a person would like Football given that they are a male, i.e., $P(Football/Male)$.To calculate this using the conditional probability formula, we divide the joint probability of "the person is male and likes football" by the probability of "the person is male." Hence, $P(Football|Male) = P(Male,Fotball)/P(Male) = 0.24/0.54 = 0.45$.

The Marginal Probability is the probability of an event irrespective of the outcome of another variable. In the figure above, $P(female)$ or $P(rugby)$, , for example,, is a marginal probability. Here, $P(female)$ doesn't depend on which sport a female plays.

Bayes Theorem:

In probability theory, Bayes’ theorem or Bayes’ rule is a mathematical formula used to calculate the conditional probability of events. Essentially, Bayes’ theorem describes the probability of an event based on prior knowledge of the conditions that might be relevant to the event.

Mathematically, it can be written as $P(A|B)=P(B|A)*P(A)/P(B)$. Note that the Bayes formula is valid only when A and B are independent events, i.e., A's outcome is independent of B and vice-versa.
In the example shown above, $P(Male|Football)$ can be calculated using the below-mentioned formula.

Parameter Estimation:
The parameter estimation technique is used to estimate the parameters randomly from a given sample distribution data. The below figure shows different parameter estimation techniques.

Learning each of these techniques is out of the scope of the article. So instead, this article will cover the maximum likelihood estimation technique for parameter estimation.

Maximum Likelihood Estimation:
Maximum Likelihood Estimation(MLE) is a parameter estimation technique that maximizes the likelihood of getting the data we observed. For applying Maximum Likelihood Estimation(MLE), data must satisfy iid assumptions, i.e.,

1. The observation of any given data point does not depend on any other data point (each gathered data point is an independent experiment).
2. Each data point is generated from the same distribution family with the same parameters.

This estimation is done following three steps:

• Find the likelihood function for the given random variables $(X1, X2, ... , Xn)$.
• Maximize the likelihood function concerning θ.
• Finally, find the value of θ.

Naive Bayes Classifier:
The naive Bayes Classifier is a probabilistic supervised classifier based on Bayes Theorem. The Naive Bayes algorithm is constituted of two words, Naive and Bayes, Which can be described as:

Naive: The classifier assumes that the occurrence of a specific feature is independent of the occurrence of other features. Suppose a car from one particular company is identified based on tag, colour, and height. Here each feature individually contributes to identifying that car without depending on each other.

Bayes: It depends on the principle of Bayes' Theorem.

Example:
Let's look at the figure below.

Problem: Players will play if the weather is rainy. Is this statement correct?
Answer: $P(Yes | rainy) = P( rainy| Yes) * P(Yes) / P (rainy)$
Here we have $P (rainy|Yes) = 2/9$, $P(rainy) = 5/14$, $P(Yes)= 9/14$. Now, $P (Yes | Sunny) = 2/9 * 9/14 / 5/14 = 0.40$, which is low probability.

Pros and Cons of Naive Bayes Classifier:

• The naive Bayes classifier is easy to implement; it is fast and can work for both multiclass and binary classification.
• It is the most popular choice for text classification problems.
• Naive Bayes assumes that all features are independent or unrelated, so it cannot learn the relationship between features.
• Also, this independence assumption rarely holds for real-life scenarios.
• Suppose a categorical variable has a category in test data that was not observed in the training data. In that case, the model will assign a 0 (zero) probability and will be unable to make a prediction. The phenomenon is called “Zero Frequency.” To solve this, a smoothing technique is used. One of the most straightforward smoothing techniques is called Laplace estimation.