Intuition behind the use of Eigen vectors in PCA

Discussion on Quantitative Methods in Finance
Forum rules
DISCLAIMER: We do not warrant or represent that this forum or its content is free of viruses, worms or other code that might be contaminating or destructive. We cannot guarantee that documents or files downloaded from the Site will be free from viruses and we do not accept any responsibility for any damage or loss caused by any virus. Accordingly, for your own protection, you must use virus-checking software when using the forum. You must not post or provide to us via the forum, any document or file which you believe may contain a virus. You must virus check any document or file which you intend to post or provide to us via the forum. You must ensure that any document or file you intend to post to the forum does not contravene any applicable laws or contravene any person's legal rights. We do not accept any responsibility for any damage or loss you may suffer.
ninjapunit
Posts: 20
Joined: Fri Sep 16, 2011 5:12 am

Intuition behind the use of Eigen vectors in PCA

Postby ninjapunit » Sun Aug 05, 2012 5:03 pm

Hi Carol,

I read in some texts that the main objective of PCA is to choose an orthonormal basis from which most of the data can be explained. Essentially, this is achieved through applying a rotation matrix R to the initial data set hence reducing its no. of dimensions. What I miss is the intuitive idea behind the eigenvectors of the covariance/correlation matrix being this rotation matrix R. I understand the equations reading the volume I, but I am more looking at the intuitive aspect of the things.

Can you kindly guide?

Thanks!
Puneet

ninjapunit
Posts: 20
Joined: Fri Sep 16, 2011 5:12 am

Re: Intuition behind the use of Eigen vectors in PCA

Postby ninjapunit » Tue Aug 07, 2012 7:51 am

1. Also, I get that mathematically, while trying to maximize the variance explained, we stumble on to the eigenvectors of the covariance matrix.
2. But, there is also this fact that this is the same matrix that covariance matrix can't rotate.

So, I am trying to connect these two sentences and draw an intuitive understanding of how should 2 lead to 1? Or am I thinking a bit too much?? :)

coalexander
Posts: 815
Joined: Sun Sep 28, 2008 10:30 pm

Re: Intuition behind the use of Eigen vectors in PCA

Postby coalexander » Tue Aug 07, 2012 8:06 am

Hi Puneet

Thanks for your further post, but I am still not sure that I understand what it is that you are asking....hence the delay to answer your question. Is it "Why is the matrix of eigenvectors a rotation matrix?"...?

ninjapunit
Posts: 20
Joined: Fri Sep 16, 2011 5:12 am

Re: Intuition behind the use of Eigen vectors in PCA

Postby ninjapunit » Tue Aug 07, 2012 11:30 am

So, geometrically speaking, when we multiply C = A X B , we are switching to a new axes system, with each row of A being the new axes and the resultant matrix C is how B would get expressed in this new system. Is my understanding right?

If yes, then similarly when we express principal components as P = W X it means that we are rotating our axes system to align along a matrix W (we will assume we don't know what it is)

Now, I understand from reading another texts that we should be aligning to the axes which has maximum variance, then the one perpendicular to it which explains the max of the remaning variance, so on and so forth.

Mathematically, it turns out that this matrix W is nothing but our good old eigen vector of covariance matrix. So, rather than working this top down using mathematical equations, I was wondering if we could use our intuitive understanding of eigen vectors to have some sort of intution as to why this is so.

I hope I am making some sense. :)

ninjapunit
Posts: 20
Joined: Fri Sep 16, 2011 5:12 am

Re: Intuition behind the use of Eigen vectors in PCA

Postby ninjapunit » Tue Aug 07, 2012 11:35 am

Also, the reason I ask this is, I have to give a training on PCA to a bunch of finance guys who are from varied backgrounds- commerce, arts included. My objective is to give an intuitive understanding of concepts which they can visualize and relate to. I am a big fan of the idea that the true test of our knowledge is our ability to explain it to lay-men. Like you do all the time on this forum :)

coalexander
Posts: 815
Joined: Sun Sep 28, 2008 10:30 pm

Re: Intuition behind the use of Eigen vectors in PCA

Postby coalexander » Tue Aug 07, 2012 11:56 am

Yes, you are shirting to new coordinates ...... this is because a matrix is a linear transformation -- take a look at I.2.3.1 -- maybe this is the bit of intuition you need?

An orthogonal matrix is one that has orthogonal columns, eg if it is a 2x2 matrix, the column vectors are at right angles in the plane. So eigenvectors are at right angles, but they are not along the x and y axis. But given two new vectors at right angles, you can rotate the plane so that these become the axis.

Orthogonal matrices are typically made up of rotations (also, reflection matrices and permutation matrices are orthogonal but even they can be thought of as made up of elementary rotations)

Hope these comments help your intuition and good luck with your course.

Carol

ninjapunit
Posts: 20
Joined: Fri Sep 16, 2011 5:12 am

Re: Intuition behind the use of Eigen vectors in PCA

Postby ninjapunit » Tue Aug 07, 2012 12:13 pm

Thanks Carol. But, why eigen vectors of Covariance matrix? Why not some other orthogonal matrix? What's so special about the eigen vectors? I am looking for an intuitive understanding of this part.

coalexander
Posts: 815
Joined: Sun Sep 28, 2008 10:30 pm

Re: Intuition behind the use of Eigen vectors in PCA

Postby coalexander » Tue Aug 07, 2012 1:27 pm

Covariance matrix is not orthogonal, the eigenvector matrix is. You can also use correlation matrix -- see Volume II chapter 2 for a full discussion.

The rationale for using covariance matrix described on page 65-66 and see (I.2.37) in particular,

Cheers, Carol

ninjapunit
Posts: 20
Joined: Fri Sep 16, 2011 5:12 am

Re: Intuition behind the use of Eigen vectors in PCA

Postby ninjapunit » Tue Aug 07, 2012 1:54 pm

Sorry, couldn't make myself clear once again. Why "eigenvectors"? :)

Any intuitive understanding for the same?

coalexander
Posts: 815
Joined: Sun Sep 28, 2008 10:30 pm

Re: Intuition behind the use of Eigen vectors in PCA

Postby coalexander » Tue Aug 07, 2012 2:05 pm

German word Eigen means 'real'....stems from similarity transforms (which diagonalize a matrix) and the Eigenvector matrix is one such transform. Cheers, Carol


Return to “Volume I”

Who is online

Users browsing this forum: No registered users and 1 guest