I'm designing an experiment to evaluate reagents used to develop fingerprints. There will be a variety of objects with different surfaces, 2 different reagents, 3 different fingerprint donors (prints deposited intentionally on the object), and 3 human graders. Photos will be taken of the developed prints on the objects for the grader to visually evaluate.
I've created a 5 level visual standard, similar to a Likert scale, to use for grading both the print development and the background development. High contrast (strong print development with minimal background development) is the ideal situation. For the visual standard, there are separate scales (0-4) for the print and the background, with visual representations of each level for both.
From what I understand, the data generated from this experiment would be discrete. I'm unsure whether it would be considered ordinal, interval, or ratio. It seems like it would have more information than just a simple ranking (for ordinal) because of using the visual standard. I think it makes sense to say there is an “absolute zero” because there will be examples of no print development and no background development. Does this mean the data would be considered a ratio subtype? Or is there another datatype that I’m not aware of that is more appropriate?
I’m also unsure of how to treat the dual grading scale results (for the print and the background). This doesn’t seem like the classic “before” and “after” medical study scenario of paired data. But I’m not sure if the dual grades would be considered “paired” in a different sense. Perhaps I need to make a single “combined” grade using the ratings for the print and the background, but I’d like to keep them independent, if possible.
The most important question to answer for the experiment will be whether there is a statistically significant difference between the two reagents (which reagent produces the highest contrast prints). I’m assuming the data will probably not be normally distributed, so I’ll need to use a nonparametric test. I don’t have a statistical software package to use, but I do have Excel.
I’d appreciate hearing recommendations for the type of hypothesis test to use (with the consideration of doing the calculations with Excel), a rough estimate of how many replicates I’d probably need, how to interpret the results of the hypothesis test , and anything else that is important to consider to draw meaningful conclusions.
Best Answer
I would agree with most of what @Emma Jean says but a few extra points to consider:
1) Regression models are actually quite robust to things like the Likert scale, so you don't need to rule them out - see https://link.springer.com/article/10.1007/s10459-010-9222-y. However, I am not suggesting you use this.
2) What many in sensory science do is give people a line of set length with no number (see this). One end of this line would be perfect print development and the other end would be absolutely no print development. You then have the observer mark along this line where they think the current sample lies and repeat for your other measure background development. You can then simply use a ruler to measure how far along the line the value is, thereby creating a continuous, albeit bounded, measure which conforms (close enough) to the assumptions.
3) If you are going to have two measures (print and background development) there are two ways forward:
4) The final thing you need to consider is practical significance. Statistical significance is all well and good but it tells you nothing about whether the difference is actually meaningful in the context of the real world. This is something that you have to decide on using your subject matter knowledge to pre-define what "size" difference has practical meaning.
Once you have figured out the effect size you want to be able to detect you can use a power analysis (there are free online generators for this that you just enter the value you have!) and it will tell you how big a sample you need.