I came up with an idea to use machine learning for automatic grading of topic-specific texts.
More specifically, I will first use normal text classification techniques to sort all candidate texts into topics. Then, I want to be able to judge the quality of texts in specific topics:
For instance, news articles in different topics (Technology, Sports, International, Business, etc). Firstly, every article will be sorted into one topic. And what I want to archive then is to automatically grade this 'Technology' article based on the range of 1-10 or good/poor.
The criteria for such scoring scheme may be:
- Elaboration (rich in details)
- Coverage of all key words
- Length?
(It is actually a little bit hard for me to quantify my criteria. The grading criteria here can be very similar to those used when human beings instinctively and quickly judge quality of two articles on a same story.)
However, as I reckon, text classification based on 'bag of words' may not be able to perform this task well. (Or will it?)
I believe such task is well-researched, and even it is a whole field rather than text classification. But I haven't figured out how to do this by searching around. Could you please refer me to related techniques/discussions/names or give me some hints if text classification can handle that.
Many thanks in advance.
Best Answer
There are two basic components, technical analysis (grammar, sentence length, etc) and machine learning/statistical analysis.
The best (IMHO) papers on this are those involving automated essay grading. It's a little dated, but this paper covers the techniques employed by all of the major vendors.
ETS (the people behind the TOEFL and the GRE) have put a LOT of work into this area, and they are surprisingly open about their research. The wall they run up against is that of length and specificity. They can only grade papers (with a high degree of accuracy) that are short and have specific prompts.
The biggest problem I see is that ALL of these grading facilities are based on machine learning. Even ETS uses their graders as backups for human graders; they pay 1 human to grade and if their grade doesn't match the ML grade, the essay is handed to another human grader. It essentially cuts their costs in half but it doesn't replace human graders.
If you really want to try analysis on ad-hoc text you will probably need to use the API of someone that has access to a lot of unstructured text, like OpenCalais or AlchemyAPI or even Zamenta for the categorizing and then roll your own quality analysis. However, that's a bit out of my pay range.
update Jonny's answer is really not getting enough credit. If you are really concerned about content quality, analyzing network connections and other measurements of human behavior are REALLY valuable. Again, I am no expert in this field!
It would help if you could tell us what our accuracy/validity requirements are.