classificationnatural languageprecision-recalltext-summarization
As you know extractive text summarization is a binary classification problem!(a sentence should be included in summary or not). we have developed our text summarization system with three different algorithms and evaluated them with ROUGE. here is our result
My question is which parametes is more important in text summarization systems. As you can see model 1 has higher recall but lower precision and model 2 has lower recall but higher precision so which one should be selected as the best model?
Users often prefer higher precision to higher recall. For example, you (and Google) really want the first 15 or 30 results for a search engine query to be accurate (=high precision), but neither of you are particularly concerned if you miss one or two of the millions on pages on the web (=low recall). Obviously, this depends a bit on the application, but I would imagine summarization is similar.
The bad news is that rapid miner has apparently computed precision and recall for each class separately, and it looks like your recall is quite bad (~13%) on the in-summary class, which is presumably more important than the not-in-summary sentences.
I suspect that this is because you have fairly unbalanced classes--it looks like there are ~3x more not-in-summary examples in your data set. You could potentially tweak the logistic regression's decision threshold (probably 0.5 by default) to give extra weight to the positive examples.
As a user of these methods I need to gauge how far I can rely on the algorithms and how far I need to use humans to do some post-processing on the summarisations.
How "good" is a particular absolute ROUGE score? I'm defining "good" as "minimises the need for human post-processing".
There are two aspects that may impact the need for human post-processing:
ROUGE doesn't try to assess how fluent the summary: ROUGE only tries to assess the adequacy, by simply counting how many n-grams in your generated summary matches the n-grams in your reference summary (or summaries, as ROUGE supports multi-reference corpora).
If there are multiple references, the ROUGE-1 scores are averaged. Because ROUGE is based only on content overlap, it can determine if the same general concepts are discussed between an automatic summary and a reference summary, but it cannot determine if the result is coherent or the sentences flow together in a sensible manner. High-order n-gram ROUGE measures try to judge fluency to some degree. Note that ROUGE is similar to the BLEU measure for machine translation, but BLEU is precision- based, because translation systems favor accuracy.
Note that BLEU has the same issue, as you can see on these correlation plots, taken from {1}:
What is the best way to really understand what a ROUGE score actually measures?
In short and approximately:
ROUGE is more interpretable than BLEU (from {2}: "Other Known Deficiencies of Bleu: Scores hard to interpret"). I said approximately because the original ROUGE implementation from the paper that introduced ROUGE {3} may perform a few more things such as stemming.
References:
Best Answer
This is extractive text summarization.
I assume you meant model 3
That depends on the user needs: sometimes it is preferable to have a bit too many sentences that somehow overlap, sometimes it is preferable to have a short summary potentially skipping important sentences.