Factor analysis is a statistical method. It is used to describe variability among correlated observed variables in terms of a potentially lower number of unobserved variables.

The generative model is given by

**y = μ + Λx +ε**

**y** is P ×1 dimension observed variable

**μ** is P ×1 dimension mean vector

**Λ** is P × R dimension factor loading matrix

**x** is R×1 dimension unobserved variable (or latent variable)

**ε** is P ×1 dimension error term

We assume that

**Ε(x) = Ε(ε) = 0**

**Ε(ΛΛ ^{T} ) = Ι
**

**p(x) = N (x |0, I )**

**Ε(y) = μ**

**Σ = Ε(yy ^{T} ) = ΛΛ^{T} + Ψ**

**p(y|θ ) = N(y|μ, ΛΛ**

^{T}+Ψ)Mean **μ** is estimated using the observed variable.

The model parameters **Λ,Ψ** are estimated using **expectation maximization (EM)** algorithm.

Initially, model parameters **Λ,Ψ** are selected with random values and iteratively updated with EM algorithm.

**In E step**, the posterior **p(x ^{n} |y^{n}, θ^{t})** is defined as below,

**q**

^{t}^{+1}= p(x^{n}|y^{n}, θ^{t}) = N(x^{n}|m^{n},V^{n})**V**

^{n}= (I −Λ^{T}Ψ^{−1}Λ)^{−1}**m**

^{n}= V^{n}Λ^{T}Ψ^{−1}(y^{n}− μ )**In M step**, the model parameters **Λ ^{t+1}, Ψ^{t+1}** are updated with posterior estimation as below,

**Λ**

^{t+1 }=( Σy^{n}m^{nT})(ΣV^{n})^{-1}**Ψ**

^{t+1}= 1/N diag (Σy^{n}y^{nT}+ Λ^{t+1 }Σm^{n}y^{nT})In each iteration, likelihood estimate **L(Λ,Ψ)** is estimated in order to confirm whether model is converged. Likelihood estimate **L(Λ,Ψ)** is estimated as below,

**L(Λ,Ψ) = N/2 log|Ψ| − N/2 tr(SΨ ^{−1})**

**Where S = 1/N Σ (y ^{n}-Λx^{n})^{T}(y^{n}-Λx^{n})**

The maximum likelihood estimate values are plotted again iterations, and it can be observed that when the number of iterations increases, the model is converged as shown in following Figure,

The main applications of factor analysis is to reduce the number of variables and to detect structure in the relationships between variables. Factor analysis is commonly used to model the varialbiy in speaker and face recognition applications.