Principal Component Analysis (PCA)

Is a technique commonly used in machine learning for data complexity reduction. Based on the concept of projections, the goal is to reduce the number of total variables used in the analysis.

• Leads to smaller datasets while minimizing information loss.
• Turn data easier to be visualized.

The main idea here is to project the data into the directions with most variance. Combining the concepts of eigenvalues and eigenvectors, projections and covariance matrix is possible to find these directions. The covariance matrix $C$ characterizes the spread of the data, it’s eigenvectors tells the direction in which the matrix can be viewed as just a straight stretching and the largest eigenvalue tells which stretching is greatest.

Note

Larger the eigenvalue larger the variance when projecting data.

Mathematical Formulation

1. Given a dataset matrix ${\mathbf{X}}_{n\times m}$ where $n$ is the number of observations and $m$ is the number of variables (features).
2. Center the data, calculating $\mathbf{X}-\mu$
3. Calculate the covariance matrix $C=\frac{1}{n-1}(\mathbf{X}-\mu {)}^T(\mathbf{X}-\mu )$
4. Calculate the eigenvalues and eigenvectors for the matrix $C$ and sort them from largest to smallest.
5. Create the projection matrix $(k=2)$:

$$V=\left[\begin{array}{cc}\frac{\overrightarrow{v_1}}{\Vert \overrightarrow{v_1}{\Vert }_2}& \frac{\overrightarrow{v_2}}{\Vert \overrightarrow{v_2}{\Vert }_2}\end{array}\right]$$

6. Project centered data ${\mathbf{X}}_{\mathbf{PCA}}=(\mathbf{X}-\mu )\mathbf{V}$
7. Reconstructing the information $\mathbf{X}={\mathbf{X}}_{\mathbf{PCA}}\cdot {\mathbf{V}}^T$