Factor Analysis is another powerful tool to understand what your data mean particularly when you have many variables. What Factor Analysis does is to try to find hidden variables which explain the behavior of your observed variables. Our interests here also lie in reducing the number of variables. So, we hope that we can find a smaller number of new variables which explain your data well. In this sense, it sounds very similar to PCA. Although the outcome is very similar in terms of reducing the number of variables, the approach to reduce the number of variable is different. I will explain this in the next section.
If you are a little more knowledgeable, you may have heard of the terms like Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA). EFA means that you don't really know what hidden variables (or factors) exist and how many they are. So you are trying to find them. CFA means that you already have some guesses or models for your hidden variables (or factors) and you want to check whether your models are correct. In many cases, your Factor Analysis is EFA, and I explain it in this page.
We are going to use a similar example in PCA. Let's say you have some data like this from your survey about what is important when they decide which computer to buy.
| Participant | Price | Software | Aesthetics | Brand | Family | Friend |
|---|---|---|---|---|---|---|
| P1 | 6 | 5 | 3 | 4 | 7 | 6 |
| P2 | 7 | 3 | 2 | 2 | 2 | 3 |
| P3 | 6 | 4 | 4 | 5 | 5 | 4 |
| P4 | 5 | 7 | 1 | 3 | 6 | 7 |
| P5 | 7 | 7 | 5 | 5 | 2 | 1 |
| P6 | 6 | 4 | 2 | 3 | 4 | 5 |
| P7 | 5 | 7 | 2 | 1 | 1 | 4 |
| P8 | 6 | 5 | 4 | 4 | 7 | 5 |
| P9 | 3 | 5 | 6 | 7 | 3 | 4 |
| P10 | 1 | 3 | 7 | 5 | 2 | 4 |
| P11 | 2 | 6 | 6 | 7 | 6 | 5 |
| P12 | 5 | 7 | 7 | 6 | 7 | 7 |
| P13 | 2 | 4 | 5 | 6 | 6 | 2 |
| P14 | 3 | 5 | 6 | 5 | 2 | 3 |
| P15 | 1 | 6 | 5 | 5 | 4 | 5 |
| P16 | 2 | 3 | 7 | 7 | 5 | 6 |
For successfully doing Factor Analysis, we need more data than this example. If your want to find n factors, you want to have roughly 3n - 4n dimensions of data, and 5n - 10n samples. And Factor Analysis assumes the normality of the data, so it is not a great tool for ordinal data. However, in practice, we can use Factor Analysis on ordinal data if the scale is 5 or more, and data can be treated as interval data.
Through Factor Analysis, you want to find hidden variables (common factors) which may explain the responses you gained. For looking at how to do Factor Analysis in R, I would like to briefly explain the difference between PCA and FA.
The intuition of Principal Component Analysis is to find new combination of variables which form larger variances. Why are larger variances important? This is a similar concept of entropy in information theory. Let's say you have two variables. One of them (Var 1) forms N(1, 0.01) and the other (Var 2) forms N(1, 1). Which variable do you think has more information? Var 1 is always pretty much 1 whereas Var 2 can take a wider range of values, like 0 or 2. Thus, Var 2 has more chances to have various values than Var 1, which means Var 2's entropy is larger than Var 1's. Thus, we can say Var 2 contains more information than Var 1.
Although the example above just looks at one variable at one time, PCA tries to find linear combination of the variables which contain much information by looking at the variance. This is why the standard deviation is one of the important metrics to determine the number of new variables in PCA. Another interesting aspect of the new variables derived by PCA is that all new variables are orthogonal. You can think that PCA is rotating and translating the data such that the first axis contains the most information, and the second has the second most information, and so forth.
The intuition of Factor Analysis is to find hidden variables which affect your observed variables by looking at the correlation. If one variable is correlated with another variables, we can say that these two variables are generated from one hidden variable, so we can explain the phenomena with that one hidden variable instead of the two variable. Let's take a look at the correlation matrix of the data we have (see the code example below to create the data frame) before doing Factor Analysis.
And you get the correlation matrix.
So, it looks like that Price has strong negative correlations with Aesthetics and Brand, and Friend has a strong correlation with Family. This means that we can expect that we will have two common factors, and one will be related to Price, Aesthetics, and Brand, and the other will be related to Friend and Family. Let's move on to Factor Analysis, and see what will happen.
In the following code example, I skipped some details, such as using varimax rotation or promax rotation (R uses varimax rotation by default). If you want to know more details, I recommend you to read other books or references for now. I may add these details later, but not sure…
First, we prepare the data.
Factor Analysis is easy to do in R. Let's do Factor Analysis assuming that the number of the hidden variables is 1.
And you get the result.
Here, the factor analysis is doing a null hypothesis test in which the null hypothesis is that the model described by the factor we have found predicts the data well. So, we have the chi-square goodness-of-fit, which is 12.8, and the p value is 0.17. This means, we cannot reject the null hypothesis, so the factor predicts the data well from the statistics perspective. This is why the result says “Test of the hypothesis that 1 factor is sufficient.” Let's take a look at the Factor Analysis with two factors.
The p value gets larger, and the Cumulative portion of variance becomes 0.61 (with one variable, it is 0.37). So the model seems to be improved. Loadings shows the weights to calculate the hidden variables from the observed variables.
But obviously, the model gets improved if you have more variables, which shows the trade-off between the number of variables and the accuracy of the model. So, how should we decide how many factors we should pick up? This is the topic for the next section.
We found the two factors in the example, which are:
| Factor 1 | Factor 2 | |
|---|---|---|
| Price | -0.657 | |
| Software | -0.161 | 0.119 |
| Aesthetics | 0.933 | |
| Brand | 0.928 | 0.242 |
| Friend | 0.100 | 0.992 |
| Family | 0.620 |
In the results of FA, some coefficients are missing, but this means these coefficients are just too small, and not necessary equal to zero. You can see the all coefficients by doing like fa$loadings[,1] with more precisions.
Although the goodness-of-fit tells you whether the current number of variables are sufficient or not, it does not tell whether the number of variables are large enough for describing the information that the original data have. For instance, why don't we try three factors instead of one or two factors? There are a few ways to answer this question.
This means whether you can explain your new variables in a sensible way. For example, Factor 1 has large weight on Price, Aesthetics and Brand, which may indicate whether people want practical aspects or fashionable aspects on their computers. Factor 2 has large weights on Friend and Family, which seems to mean that people around users have some effects on the computer purchase. Thus, both factors seem to have some meanings, and that's why we should keep them.
This is not really a mathematical way to determine the number of factors, but is a standard way to do. Because we want to find factors which explain something, we can just ignore factors which don't really make sense. This is probably intuitive, but I know you may argue that it is too subjective. So, we have more mathematical ways to determine the number of factors.
Similar to PCA, you can look at the cumulative portion of variance, and if that reaches some numbers, you can stop adding more factors. Deciding the threshold for the cumulative portion is kind of heuristic. It can be 80% similar to PCA. If your focus is on reducing the number of variables, it can be 50 - 60 %.
The Kaiser rule is to discard components whose eigenvalues are below 1.0. This is also used in SPSS. You can easily calculate the eigenvalues from the correlation matrix.
So, we can determine that the number of factors should be 2. One problem of Kaiser rule is that it often becomes too strict.
Another way to determine the number of factors is to use Scree plot. You plot the components on the X axis, and the eigenvalues on the Y axis and connect them with lines. You then try to find the spot where the slope of the line becomes less steep. So, how exactly should we find the spot like that? Again, it is kind of heuristic. In some cases (particularly when the number of your original variables are small like the example above), you can't find a clear spot like that (try to make a plot by using the following code). Nonetheless, it is good to know how to make a Scree plot.
The following procedure to make a Scree plot is based on this webpage. You also need nFactors package.
Unfortunately, there is no definitive way to determine the number of factors. You probably try to do multiple ways explained here, and see how many factors each of them is suggesting. They should suggest you more or less the same number of factors, so you can have a pretty good idea of how many factors you should choose.