Feature Extraction in Machine Learning

Feature Extraction -definition

 Given a set of features F = {𝒳1,.....,𝒳N}
   the Feature Extraction ("Construction") problem is to  map F to some feature set F" that maximizes the learner's ability to classify patterns.

Find a projection matrix w from N-dimensional to M-dimensional vectors that keeps error low.

Assume that N features are linear combination of M < N vectors 
    
    Zi = Wi1𝒳i1 + ......+ Wid𝒳iN
         Z = Wt𝒳

What we expect from such basis
    - Uncorrelated cannot be reduced further
    - Have large variance or otherwise bear no information.

Algebraic definition of PCs

 PCA

PCA for image Compression 

Is PCA a good criterion for classification ?

- Data variation determines the projection direction
- What's missing ?
       - Class information 


What is a good projection ?

- Similarly, what is a good criterion ?
    - Separating different classes

What class information may be useful ?

Between-class distance
  - Distance between the centroids of different classes
Within-class distance
  - Accumulated distance of an instance to the centroid of its class.
Linear discriminant analysis (LDA) finds most discriminant projection by
  - maximizing between-class distance
  - and minimizing within-class distance

Linear Discriminant Analysis

Find a low-dimensional space such that when 𝓍 is projected, classes are well-separated.

Means and Scatter after projection

Good Projection

- Means are as far away as possible
- Scatter is small as possible
- Fisher Linear Discriminant
      J(w) = (m1 - m2)2 /s12 + s2 square