My survey is based on observations and interviews. The observation questions are yes/no type questions and interview questions are 4 points.
$1=$ never, $2=$ sometimes, $3=$ usually, and $4=$ always.
Yes $=1$, No $=0$.
If I add up all these with highest possible score and divide it by the total score received, I can then retrieve a total score for the survey.
For example, let's say I have 5 responses for the Likert and 5 for the yes/no. $5\times4$ $=20$ (with 4 being the highest possible score you can get on the Likert scale)
and $5\times1$ $=5$ $(1$ being the highest possible score for yes/no question). I then add up both to find the highest possible score $=25$. If I get a combined score of $20$, do I divide it by the possible score, $20\div25=80$%? Can I build my questionnaire and calculations like this?
My main objective is to get one percentage / score for the entire questionnaire. I thought of accomplishing this in 2 ways:
- Get the sum of earned score from both scales divided by total possible scores for both scales.
- Get a percentage by using the above calculation for individual scales, which will give me two sets of percentages, and then take an average for those percentages.
I am not sure which way I should go with, or if this even makes sense doing it this way.
Best Answer
This answer is based on another answer of mine, but adapted to your question slightly.
If you're sure your items are all measuring the same latent construct, you could use a partial credit model to account for differences in response scaling across all items. If the items with four-point Likert scale (polytomous) measurements are all on the exact same scale though, you might be better off using a rating scale model of the polytomous items and a separate, probably more basic item response theory model for the binary items. John Michael Linacre and Benjamin D. Wright posted some discussions of the differences between partial credit and rating scale models over at rasch.org that might give you a better sense of what you'd be dealing with if you go the item response theory route here.
Some latent variable analysis programs will let you set certain thresholds to be equal across certain items and leave another item's threshold freely estimated. You might be able to blend the partial credit and rating scale models this way by setting your polytomous items' thresholds (each item will have three) to be equal across items, and estimating the binary items' single threshold independently of the polytomous items. Depending on your theory about the binary items, they could all have the same threshold as each other, or they could each have their own, or maybe somewhere between those two extremes...but I'm not exactly sure this is all you'd need to do to have the best of both worlds.
The simple, "classical test theory" approach that weighs every item equally would probably have you just standardize all the items and average the $z$-scores, but I don't think that's a good idea, because four-point Likert scales may not approximate a continuous dimension well enough (and a binary item definitely won't; it might not even make sense), though the average of 12 polytomous items might be approximately continuous enough. I've seen it suggested that each item's Likert scale should have at least five items to approximate a continuous distribution, and at least five Likert scale items should measure the same scale if their simple sum / average is to approximate a continuous dimension. (Can't remember where, but I can look it up and edit it in if you want a source but can't find one yourself; just comment!)
If you're not sure your items are all measuring the same latent construct, I'm afraid you have other things to worry about; see these questions: