Category: SAS

The goal of this workshop and blog post is to review 3 different multivariate analyses.  We will use one common dataset to showcase the different purposes of the analyses and to showcase the different PROCedures available in SAS to conduct each analysis.

The dataset we will be using is the Fisher Iris dataset (1936), originally collected by Dr. E. Anderson and used to derive discriminant analysis by Dr. Ronald Fisher.  The dataset contains measures of petal length, petal width, sepal length, and sepal width on 50 plants of 3 varieties of Iris’.  The dataset is available within the SAS Help.  To access this dataset you will need to use the dataset name:  sashelp.iris.

Exploratory and Explanatory Analyses

When we think of statistics, most of us tend to think of our traditional hypothesis-driven analysis.  So, the ANOVAs, regressions, means comparisons, and the list goes on.  These are types of Explanatory Analyses.  There is another world of statistics, referred by some as Exploratory Analyses – those analyses that are not driven by a hypothesis.  Exploratory analyses are used more for describing relationships among variables or measures that were taken during a trial or in a dataset.  Principal Component Analysis or PCA, and Cluster Analysis are two examples of exploratory analyses, whereas discriminant analyses falls into the explanatory analysis bucket.

Principal Component Analysis (PCA)

Please review the PCA blog post for more details regarding this analysis.  This post will not provide the same level of detail but will form the basis of using the same dataset across three different analyses.

The roots of the PCA were found in 1901 and was developed by Karl Pearson.  Its primary role is top reduce the number of variables used to explain a dataset.  Factor analysis (FA) is a related process and has the same goal.  Many people confuse these two and tend to use the terms factor analysis and PCA interchangeably, when the two analyses are similar but not interchangeable.  I’ve listed a few of the primary differences between PCA and Factor analysis:

1. Both analyses begin with a correlation matrix.  PCA maintains the diagonal of the correlation matrix as 1’s, whereas Factor analysis replaces to provide a measures of the relationship of each variable with the others.
2. PCA – total variance among the variables is explained, FA common variance shared is the basis of the analysis
3. PCA is less complex mathematically compared to FA
4. PCA is one procedure, whereas FA is a family of procedures

SAS code and output using the IRIS dataset

/* For this workshop we will use the IRIS dataset */
/* Fisher’s dataset can be found in the SASHELP */
/* Library. The datsaet name is sashelp.iris */
/* SASHELP is the permanent SAS directory */

Proc print data=sashelp.iris;
Run;

/* Let’s get a sense of relationships that may exist */
/* in the dataset. We will use PROC SGPLOT to visualize */

Proc sgplot data=sashelp.iris;
    scatter x=SepalLength y=PetalLength; * / datalabel=species;
Run;

/* We will run the PROC PRINCOMP as we did in the */
/* previous workshop. Options we are using include */
/* plots=all to show all plots available in the PROC */
/* n = 3 – we will start without this option and */
/* and then add it back to see only the 3 components */

ods graphics on;
Proc princomp data=sashelp.Iris standard plots=all n=3;
    var SepalLength PetalLength SepalWidth PetalWidth;
Run;
ods graphics off;

To view the output.  The output explanations will be the same as the explanations reviewed in the last post – only a different dataset.

Cluster Analysis

Cluster analysis is a multivariate analysis that does not have ah hypothesis.  We are interested in seeing whether there are any natural clusters or groups of the data.  Clusters can be based on either the variables or measures collected in the dataset, OR they can be based on the observations within the dataset – variables or observations.

Clustering techniques will use two processes: distances and linkages.  Being familiar with these terms may help you to select the most appropriate clustering technique for your data.

Distance:  quantitative index defining the similarities of the clusters remaining in the analysis at each step.

Linkage: two clusters that have the smallest distance between them as determined by particular distance measures are then linked together to form a new cluster.

Standardizing the variables to be used in a cluster analysis is essential.  Clustering techniques use some measure of “distance”, ensuring that all the variables are on the same level, will ensure a better clustering.

There are 2 broad types of clustering techniques used in Cluster Analysis:

• Hierarchical
• K-means

1. Hierarchical clustering.
   Cluster or group a small to moderate number of cases based on several quantitative attributes  the groups are clustered on a given set of variables – so when we talk about these clusters we can only discuss their merits based on the variables used to create the groups.  Remember context!
2. K-means
   Creating cluster from a relatively large number of cases based on a relatively small set of variables.  K-means uses an iterative technique, cases are added to a cluster during the analysis rather than at the end – this allows some cases to shift around before the analysis is complete.  You also need to specify how many clusters you want as a result with K-means clustering.

SAS code and output using the IRIS dataset

In SAS there are 2 PROCedures that are commonly used for Cluster Analysis:

PROC Cluster and PROC Fastclus:

Directly from the SAS Online documentation:
“The CLUSTER procedure is not practical for very large data sets because, with most methods, the CPU time is roughly proportional to the square or cube of the number of observations. The FASTCLUS procedure requires time proportional to the number of observations and can therefore be used with much larger data sets than PROC CLUSTER. If you want to hierarchically cluster a very large data set, you can use PROC FASTCLUS for a preliminary cluster analysis to produce a large number of clusters and then use PROC CLUSTER to hierarchically cluster the preliminary clusters.”

When you create clusters – in any package – it is handy to calculate the means of the clusters and to run them through a Frequency analysis – essentially we want to be able to review some descriptive statistics on our new groups or clusters.  PROC Fastclus saves all of this information for us be default or as part of the PROC coding, whereas with PROC Cluster you need to add in a few extra steps.  Part of the coding that is commonly used includes a SAS Macro with PROC Cluster to run these descriptive statistics on our output clusters.

However, be assured that the output is the same whether you use PROC Fastclus or PROC Cluster with the macro.  For simplicity, we will only use the PROC Fastclus syntax for our example.

/* Cluster Analysis */
/* Creating 2 clusters, saving the results in a new dataset called CLUS */
/* Try a Proc PRINT to see what is found in the new dataset CLUS */
Proc fastclus data=sashelp.iris maxc=2 maxiter=10 out=clus;
    var SepalLength SepalWidth PetalLength PetalWidth;
Run;

/* Using the resulting dataset to get a feel for who landed in what cluster */
Proc freq data=clus;
    tables cluster*species;
Run;

/* Creating 3 clusters, saving the results in a new dataset called CLUS */
Proc fastclus data=sashelp.iris maxc=3 maxiter=10 out=clus;
    var SepalLength SepalWidth PetalLength PetalWidth;
Run;

/* Using the resulting dataset to get a feel for who landed in what cluster */
Proc freq data=clus;
  tables cluster*Species;
Run;

/* To obtain a graphical presntation of the clusters we need to run the */
/* Proc CANDISC to get the information needed for the graphical output */

Proc candisc anova out=can;
    class cluster;
    var SepalLength SepalWidth PetalLength PetalWidth;
    title2 ‘Canonical Discriminant Analysis of Iris Clusters’;
Run;

Proc sgplot data=Can;
    scatter y=Can2 x=Can1 /group=Cluster ;
    title2 ‘Plot of Canonical Variables Identified by Cluster’;
Run;

To view the resulting output.

Extra piece of SAS code.  If you need to standardize your variables before putting them into a Cluster analysis here is a sample piece of code that you can use:

/* If you need to standardize your variables – this is how you would do it */
Proc standard data=sashelp.iris out=iris mean=0 std=1;
    var SepalLength SepalWidth PetalLength PetalWidth;
Run;

/* Run a Proc PRINT to see what happened to your data and what changes happened */
Proc print data=iris;
Run;

/* Run a Proc MEANS to check whether the standardization worked or not */
Proc means data=iris;
    var SepalLength SepalWidth PetalLength PetalWidth;
Run;

Discriminant Function Analysis

As noted earlier, this analysis is not an exploratory analysis but an explanatory analysis.  In fact it is very similar to an Multivariate ANOVA or MANOVA.  It does however, have 2 distinct but compatible purposes:

1. To determine whether the characteristics used to define the groups hold true or not
2. To classify or predict the group membership of new observations based on the discriminant function.

So what does discriminant function do?  Essentially it creates a weighted linear combination of the variables used in the analysis which is then used to differentiate or group observations into groups.  Logistic regression comes to mind when you define discriminant analysis, however with logistic regression, the predictors can be quantitative or categorical and the fitted curve is sigmoidal in shape.  Discriminant analysis can only use quantitative variables and all the assumptions of a general linear model must be met.  So yes that means residual analysis – normality, homogeneity of variances, ….

One of the biggest challenges with discriminant analysis is sample size!  The smallest group in your dataset MUST exceed the number of predictor variables by a “lot”.  Papers have suggested at least 5 X or at least 10 X.

So, in the end discriminant analysis will essentially create a regression equation from your data that will “discriminate” observations into 2 groups – a variable in your dataset.  Let’s look at the example to get a better feel for this.

SAS code and output using the IRIS dataset

/* Discriminant Analysis – Fisher’s Iris Data */
Proc discrim data=sashelp.iris anova manova listerr crosslisterr;
    class Species;
    var SepalLength SepalWidth PetalLength PetalWidth;
Run;

To view the resulting output.

Conclusion

A quick review of 3 different types of multivariate analyses using SAS and the same dataset.  Each analysis has a different purpose.  Please ensure that you use the most appropriate analysis for your research question!

Many statistical procedures test specific hypotheses.  Principal Component Analysis (PCA), Factor analysis, Cluster Analysis, are examples of analyses that explore the data rather than answer a specific hypothesis.  PCA examines common components among data by fitting a correlation pattern among the variables.  Often used to reduce data from several variables to 2-3 components.

Before running a PCA, one of the first things you will need to do is to determine whether there is any relationship among the variables you want to include in a PCA.  If the variables are not related then there’s no reason to run a PCA.  The data that we will be working with is a sample dataset that contains the 1988 Olympic decathlon results for 33 athletes.  The variables are as follows:

athlete:  sequential ID number
r100m:  time it took to run 100m
longjump:  distance attained in the Long Jump event
shotput:  distance reached with ShotPut
highjump:  height reached in the High Jump event
r400m: time it took to run 400m
h110m:  time it took to run 110m of hurdles
discus:  distance reached with Discus
polevlt:  height reached in the Pole Vault event
javelin:  distance reached with the Javelin
r1500m:  time it took to run 1500m

Let’s start with a PROC CORR to review the relationships among the variables.

Proc corr data=olympic88;
Run;

By reviewing the output found here,  we can see that there are a number of significant relationships suggesting that a PCA will be a valuable method of reducing our data from 10 variables to 2 or 3 components.

There are a few different PROCedures available in SAS to conduct a PCA.  My preferred PROC is PRINCOMP – short for principal components.  Let’s start with the basic syntax:

Proc princomp data=olympic88;
var r100m longjump shotput highjump r400m h110m discus polevlt javelin                    r1500m;
Run;

The output starts with the same correlation matrix we created using the PROC CORR.  Although you’ll notice that there are no p-values available here.  Although we see the correlations we do not know whether they are significantly different from 0 or not.  We also have the Simple Statistics available – Means and Standard Deviations.

Our next table is a table of the eigenvalues for each component.  So let’s step back and talk about what this analysis is really doing for us.

Imagine my cloud of data – throw all the data in the air – there is variation due to the different events and there is variation within each event attributed to the performance of the different athletes.  Our goal with PCA is to reduce our data from the 10 events down to 2 or 3 components that represent these 10 variables (events).  Back to my cloud of data – PCA will draw a line through all the data that explains the most variation possible with that one line – or arrow if you want to visualize this.  That will be the first component.  PCA analysis will then go back to the cloud of data and draw a second line through the data that explains the next “most” variation – 2nd component, and it will continue to do this until there is no variation left.  If you have 10 variables, you will have 10 components to explain all the variation.  Each component explains a different amount of variation.  The first will explain the most, the second lesser, and so on.  PCA will provide you with eigenvalues which are translated to an amount of variation.

Now, each component will be made up of bits and pieces of the 10 variables – we will see these as the weightings within each Component or Eigenvector.  The most challenging part of PCA, will be the definition of the components.  It’s fine to say we have 2 components, but there’s more value in trying to define what the components represent.  For this, you will use the weightings within each eigenvector or component.

Now that we have a better feeling for what is happening when we run this analysis, I said earlier that our goal is to cut down from 10 variables to 2 or 3 components.  How do we do this?  During the analysis, we will see a SCREE PLOT – we want to use this as a guide to help determine how many components we will use.  Where the elbow appears in the scree plot, there is where we cut off the number of components to use.  Yes, a subjective decision.  We will also use the % variation explained by the components as a guide and aid to support our decision.

If we look at our current output – let’s first scroll down to the scree plot.  notice it shows the principal component number on the x-axis and the eigenvalue on the y-axis.  Also note that the elbow in the curve happens around the 3rd component.  If you look up at the table that shows you the proportion of the variation explained by the components we see that Component #1 explains 34%, Component #2 explains 26%, and Component #3 explains 9%.  Given that drastic drop between Component 2 and 3, I would select working with only the first 2 components.  Subjective decision!!!  But be able to back it up.  In this case, the first two explain a total of 60% of the variation seen across the 10 events, and as I look down to component #4 it also explains 9%.  So rather than trying to explain why I didn’t include component #4 and keeping component #3 when they are so  similar, I decide to cut it at 2 components.

The next challenge is trying to “define” the 2 components.  To attempt this, look at the table with the weightings for each component.

For component #1 we see a nice split of events with all the running events holding a -ve weighting and all other events a +ve weighting.  This component could be viewed as representing the running ability of the decathletes.

For component #2 the events with the lowest values are longjump, highjump, h100m – 3 jumping or height events.  It has been suggested that this component may be represented as the strength or endurance ability of the decathletes.  Very subjective again!!

This is the basis of PCA – however, in our output there is one particular output that many associate with PCA that does not accompany the default settings for PRINCOMP.  We want to see the component plots.  In SAS we need to specify these plots.

What I recommend, run the analysis as we have above to determine how many components you want to work with first.  Once you’ve decided then you will specify this in the following code to obtain the plots of interest.  In older versions of SAS you will need to turn the ODS graphics option on and off to take advantage of the advanced graphing abilities of SAS.  Please note that I have NOT tested this in SAS Studio!!!

ods graphics on;
Proc princomp data=olympic88 n=2 plots(ncomp=3)=all;
var r100m longjump shotput highjump r400m h110m discus polevlt javelin r1500m;
RUn;
ods graphics off;

The above code will result in the following output.  One of the first things you will notice is that by adding the n=2 option in the Proc statement we are telling SAS to only calculate the first 2 components.  The plots(ncomp=3)=all produces all the plots following the Scree Plot.

You can use the Component Pattern Profile plot and the component plots to help you define what the components represent.

This is a fun and very straightforward example of how to use PCA with your data.