Solved – How to score Likert scales and test a hypothesis relating two variables

likertscales

I am want to assess whether successful implementation of newly developed policies are dependent on involvement of all relevant stakeholders and middle managers.

I have a 5-point Likert scale with ten questions (items). I collected data on 85 respondents. I assigned values ('Strongly Agree'=1, 'Agree'=2, 'Undecided'=3, 'Disagree'=4, 'Strongly Disagree'=5). I have calculated percents of valid responses, and mean and standard deviation for each item. I have summed up total agree, total disagree and total undecided (neither agree or disagree). I have put these infos into graphs.

What do I do next?

Best Answer

If you designed your questionnaire correctly you have form hypotheses between relations of constructs. For example I think you have implementation success as an Y variable dependent on manager involvement. Now, you most probably have questions in your questionnaire which measure those variables somehow.

When you have the results of your questions you can make a scale (or index for that matter) to measure your construct. You can do this by using a few methods (be sure to invert negative questions in terms of scale)

When for example you measure implementation success with 5 questions. You may take the mean of those scores (again, rescale negative questions!) assuming they have equal weights for determination of the construct. Also, you could just give them weights by guesstimates.

What's more, you could extract factor scores to have a less arbitrary weighting. All of these methods have their (dis)advantages over the others.

Checking whether questions measure the same thing can be done by using reliability (Cronbach's) alpha. But be sure to know what it does.And also be sure to use other metrics, measures, tests and a healthy dose of face validity and common sense.

Finally you investigate the relationships between your constructs (measured by your developed scales) with for example correlations or regressions.

I hope this helps, good luck!

Using `irutils`

I came across this package some months ago.

As of commit 0573195c07 on Github, the code won't work with a grouping= argument. Let's go for Friday's debugging session.

Start by downloading a zipped version from Github. You'll need to hack the R/likert.R file, specifically the likert and plot.likert functions. First, in likert, cast() is used but the reshape package is never loaded (although there's an import(reshape) instruction in the NAMESPACE file). You can load this yourself beforehand. Second, there's an incorrect instruction to fetch items labels, where a i is dangling around line 175. This has to be fixed as well, e.g. by replacing all occurrences of likert$items[,i] with likert$items[,1]. Then you can install the package the way you are used to do on your machine. On my Mac, I did

% tar -czf irutils.tar.gz jbryer-irutils-0573195
% R CMD INSTALL irutils.tar.gz

Then, with R, try the following:

library(irutils)
library(reshape)

# Simulate some data (82 respondents x 66 items)
resp <- data.frame(replicate(66, sample(1:5, 82, replace=TRUE)))
resp <- data.frame(lapply(resp, factor, ordered=TRUE, 
                          levels=1:5, 
                          labels=c("Strongly disagree","Disagree",
                                   "Neutral","Agree","Strongly Agree")))
grp <- gl(2, 82/2, labels=LETTERS[1:2]) # say equal group size for simplicity

# Summarize responses by group
resp.likert <- likert(resp, grouping=grp)

That should just work, but the visual rendering will be awful because of the high number of items. It works without grouping (e.g., plot(likert(resp))), though.

enter image description here

I would thus suggest to reduce your dataset to smaller subsets of items. E.g., using 12 items,

plot(likert(resp[,1:12], grouping=grp))

I get a 'readable' stacked barchart. You can probably process them afterwards. (Those are ggplot2 objects, but you won't be able to arrange them on a single page with gridExtra::grid.arrange() because of readability issue!)

enter image description here

Alternative solution

I would like to draw your attention on another package, HH, that allows to plot Likert scales as diverging stacked barcharts. We could reuse the above code as shown below:

resp.likert <- likert(resp)
detach(package:irutils)
library(HH)
plot.likert(resp.likert$results[,-6]*82/100, main="")

but that will complicate things a bit because we need to convert frequencies to counts, subset the likert object produced by irutils, detach package, etc. So let's start again with fresh (counts) statistics:

plot.likert(t(apply(resp, 2, table)), main="", as.percent=TRUE,
            rightAxisLabels=NULL, rightAxis=NULL, ylab.right="", 
            positive.order=TRUE)

enter image description here

To use a grouping variable, you'll need to work with an array of numerical values.

# compute responses frequencies separately by grp
resp.array <- array(NA, dim=c(66, 5, 2))
resp.array[,,1] <- t(apply(subset(resp, grp=="A"), 2, table))
resp.array[,,2] <- t(apply(subset(resp, grp=="B"), 2, table))
dimnames(resp.array) <- list(NULL, NULL, group=levels(grp))
plot.likert(resp.array, layout=c(2,1), main="")

This will produce two separate panels, but it fits on a single page.

enter image description here

Edit 2016-6-3

As of now likert is available as separate package.
You do not need reshape library or detach both irutils and reshape

Solved – Item analysis for a likert-type questionnaire – item discrimination, point-biserial, factor analysis, cronbach’s alpha, other diagnostics

Your method #1 loses information by dichotomizing in two different ways. I'd instead look at each item's correlation with the sum of all the other items (in software such as SPSS this is called "Corrected Item-Total Correlation"). For #2, where you have done something close to this, you could make a case for either Spearman's or Pearson's, and they'll hardly differ since with a 1-5 per-item range there shouldn't be many extreme outliers. You'll have to establish your own threshold, I'm afraid: how exacting do you want to be? How desirable is it to preserve a large number of items for your scales? And how concerned are you about your case-to-item ratio?

As for your questions about factor analysis, yes, to build the scales based on empirical criteria can be defensible, just as it can be to do so using a priori ideas about which item belongs to which dimension. Good research will hopefully reconcile any conflicts between the two. Items with multiple high loadings are a problem if you want uncorrelated factors, something that is often unrealistic in opinion research. At a more general level, I think you have some sense that factor analysis and scale development is best seen as a largely creative process where there are many subjective decisions to be made and often much work to do in justifying them!

Best Answer

Related Solutions

Solved – Visualizing Likert responses using R or SPSS

Using irutils

Alternative solution

Solved – Item analysis for a likert-type questionnaire – item discrimination, point-biserial, factor analysis, cronbach’s alpha, other diagnostics

Related Question

Using `irutils`