Hi Carol,
I read in some texts that the main objective of PCA is to choose an orthonormal basis from which most of the data can be explained. Essentially, this is achieved through applying a rotation matrix R to the initial data set hence reducing its no. of dimensions. What I miss is the intuitive idea behind the eigenvectors of the covariance/correlation matrix being this rotation matrix R. I understand the equations reading the volume I, but I am more looking at the intuitive aspect of the things.
Can you kindly guide?
Thanks!
Puneet
Intuition behind the use of Eigen vectors in PCA
Forum rules
DISCLAIMER: We do not warrant or represent that this forum or its content is free of viruses, worms or other code that might be contaminating or destructive. We cannot guarantee that documents or files downloaded from the Site will be free from viruses and we do not accept any responsibility for any damage or loss caused by any virus. Accordingly, for your own protection, you must use viruschecking software when using the forum. You must not post or provide to us via the forum, any document or file which you believe may contain a virus. You must virus check any document or file which you intend to post or provide to us via the forum. You must ensure that any document or file you intend to post to the forum does not contravene any applicable laws or contravene any person's legal rights. We do not accept any responsibility for any damage or loss you may suffer.
DISCLAIMER: We do not warrant or represent that this forum or its content is free of viruses, worms or other code that might be contaminating or destructive. We cannot guarantee that documents or files downloaded from the Site will be free from viruses and we do not accept any responsibility for any damage or loss caused by any virus. Accordingly, for your own protection, you must use viruschecking software when using the forum. You must not post or provide to us via the forum, any document or file which you believe may contain a virus. You must virus check any document or file which you intend to post or provide to us via the forum. You must ensure that any document or file you intend to post to the forum does not contravene any applicable laws or contravene any person's legal rights. We do not accept any responsibility for any damage or loss you may suffer.

 Posts: 20
 Joined: Fri Sep 16, 2011 5:12 am

 Posts: 20
 Joined: Fri Sep 16, 2011 5:12 am
Re: Intuition behind the use of Eigen vectors in PCA
1. Also, I get that mathematically, while trying to maximize the variance explained, we stumble on to the eigenvectors of the covariance matrix.
2. But, there is also this fact that this is the same matrix that covariance matrix can't rotate.
So, I am trying to connect these two sentences and draw an intuitive understanding of how should 2 lead to 1? Or am I thinking a bit too much?? :)
2. But, there is also this fact that this is the same matrix that covariance matrix can't rotate.
So, I am trying to connect these two sentences and draw an intuitive understanding of how should 2 lead to 1? Or am I thinking a bit too much?? :)

 Posts: 815
 Joined: Sun Sep 28, 2008 10:30 pm
Re: Intuition behind the use of Eigen vectors in PCA
Hi Puneet
Thanks for your further post, but I am still not sure that I understand what it is that you are asking....hence the delay to answer your question. Is it "Why is the matrix of eigenvectors a rotation matrix?"...?
Thanks for your further post, but I am still not sure that I understand what it is that you are asking....hence the delay to answer your question. Is it "Why is the matrix of eigenvectors a rotation matrix?"...?

 Posts: 20
 Joined: Fri Sep 16, 2011 5:12 am
Re: Intuition behind the use of Eigen vectors in PCA
So, geometrically speaking, when we multiply C = A X B , we are switching to a new axes system, with each row of A being the new axes and the resultant matrix C is how B would get expressed in this new system. Is my understanding right?
If yes, then similarly when we express principal components as P = W X it means that we are rotating our axes system to align along a matrix W (we will assume we don't know what it is)
Now, I understand from reading another texts that we should be aligning to the axes which has maximum variance, then the one perpendicular to it which explains the max of the remaning variance, so on and so forth.
Mathematically, it turns out that this matrix W is nothing but our good old eigen vector of covariance matrix. So, rather than working this top down using mathematical equations, I was wondering if we could use our intuitive understanding of eigen vectors to have some sort of intution as to why this is so.
I hope I am making some sense. :)
If yes, then similarly when we express principal components as P = W X it means that we are rotating our axes system to align along a matrix W (we will assume we don't know what it is)
Now, I understand from reading another texts that we should be aligning to the axes which has maximum variance, then the one perpendicular to it which explains the max of the remaning variance, so on and so forth.
Mathematically, it turns out that this matrix W is nothing but our good old eigen vector of covariance matrix. So, rather than working this top down using mathematical equations, I was wondering if we could use our intuitive understanding of eigen vectors to have some sort of intution as to why this is so.
I hope I am making some sense. :)

 Posts: 20
 Joined: Fri Sep 16, 2011 5:12 am
Re: Intuition behind the use of Eigen vectors in PCA
Also, the reason I ask this is, I have to give a training on PCA to a bunch of finance guys who are from varied backgrounds commerce, arts included. My objective is to give an intuitive understanding of concepts which they can visualize and relate to. I am a big fan of the idea that the true test of our knowledge is our ability to explain it to laymen. Like you do all the time on this forum :)

 Posts: 815
 Joined: Sun Sep 28, 2008 10:30 pm
Re: Intuition behind the use of Eigen vectors in PCA
Yes, you are shirting to new coordinates ...... this is because a matrix is a linear transformation  take a look at I.2.3.1  maybe this is the bit of intuition you need?
An orthogonal matrix is one that has orthogonal columns, eg if it is a 2x2 matrix, the column vectors are at right angles in the plane. So eigenvectors are at right angles, but they are not along the x and y axis. But given two new vectors at right angles, you can rotate the plane so that these become the axis.
Orthogonal matrices are typically made up of rotations (also, reflection matrices and permutation matrices are orthogonal but even they can be thought of as made up of elementary rotations)
Hope these comments help your intuition and good luck with your course.
Carol
An orthogonal matrix is one that has orthogonal columns, eg if it is a 2x2 matrix, the column vectors are at right angles in the plane. So eigenvectors are at right angles, but they are not along the x and y axis. But given two new vectors at right angles, you can rotate the plane so that these become the axis.
Orthogonal matrices are typically made up of rotations (also, reflection matrices and permutation matrices are orthogonal but even they can be thought of as made up of elementary rotations)
Hope these comments help your intuition and good luck with your course.
Carol

 Posts: 20
 Joined: Fri Sep 16, 2011 5:12 am
Re: Intuition behind the use of Eigen vectors in PCA
Thanks Carol. But, why eigen vectors of Covariance matrix? Why not some other orthogonal matrix? What's so special about the eigen vectors? I am looking for an intuitive understanding of this part.

 Posts: 815
 Joined: Sun Sep 28, 2008 10:30 pm
Re: Intuition behind the use of Eigen vectors in PCA
Covariance matrix is not orthogonal, the eigenvector matrix is. You can also use correlation matrix  see Volume II chapter 2 for a full discussion.
The rationale for using covariance matrix described on page 6566 and see (I.2.37) in particular,
Cheers, Carol
The rationale for using covariance matrix described on page 6566 and see (I.2.37) in particular,
Cheers, Carol

 Posts: 20
 Joined: Fri Sep 16, 2011 5:12 am
Re: Intuition behind the use of Eigen vectors in PCA
Sorry, couldn't make myself clear once again. Why "eigenvectors"? :)
Any intuitive understanding for the same?
Any intuitive understanding for the same?

 Posts: 815
 Joined: Sun Sep 28, 2008 10:30 pm
Re: Intuition behind the use of Eigen vectors in PCA
German word Eigen means 'real'....stems from similarity transforms (which diagonalize a matrix) and the Eigenvector matrix is one such transform. Cheers, Carol
Who is online
Users browsing this forum: No registered users and 1 guest