Solved – Weka test ARFF files

weka

I am trying to classify a question type, as a "Type" . It can be classified into 6 types.
I am using an ARFF file as input to weka with 99 training entries. I have got a model in place, and it has an accuracy of 85% (obtained using cross validation), which is to my satisfaction.

Now, when I want to test this, using a new data set, I am getting :

=== Summary ===

Total Number of Instances 0
Ignored Class Unknown Instances 7

There were 7 rows in my data. All of them seem to have been ignored.

I am new to weka and cross validated, can someone please tell me what must be going wrong here?

Here is my test input file:

@relation TypeModel

@attribute Type {person,date,object,number,place,boolean}
@attribute who {no,yes}
@attribute what {no,yes}
@attribute when {no,yes}
@attribute where {no}
@attribute which {no,yes}
@attribute how {no,yes}
@attribute city {no,yes}
@attribute state {no,yes}
@attribute country {no,yes}
@attribute '?' {no,yes}
@attribute day {no,yes}
@attribute is {no,yes}
@attribute people {no,yes}
@attribute person {no}
@attribute president {no,yes}
@attribute 'how many' {no,yes}
@attribute 'give me all' {yes,no}

@data
?,no,no,no,no,no,no,no,no,no,no,no,no,no,no,no,no,yes
?,no,no,no,no,no,yes,no,no,no,no,yes,no,no,no,no,no,no
?,yes,no,no,no,no,no,no,no,no,yes,no,no,no,no,no,no,no
?,no,no,no,no,yes,no,no,no,no,yes,no,no,no,no,no,no,no
?,no,no,no,no,no,yes,no,no,yes,yes,no,no,no,no,no,yes,no
?,no,no,no,no,yes,no,yes,no,no,yes,no,no,no,no,no,no,no
?,no,no,no,no,no,no,no,no,no,yes,no,yes,no,no,yes,no,no

Best Answer

Type is the first element in your arff file for each row. You want to classify according to that but they all have '?', unknown values. Since weka don't know what they are, it can't give you a result of classification.

Related Solutions

Solved – Weka: how to score an arff file using a NaiveBayes model

First I recommend reading about Naive Bayes.

Why does score 0.55 correspond to 0 (instance#2) but a smaller score 0.519 correspond to 1 (instance#7)?

NaiveBayes decides according to values of two probability predictions not according to if they are near to one. For example class A probability is 0.1 and class B probability 0.12 then class B is the prediction. Here score means posterior probability of class given prior probabilities of features. In the above stack overflow example: Posterior probabilities (scores) was 1/20 and 1/60. Higher one is chosen.

Where is the output documented?

You may look to class source file. But not everything in weka is well documented. This output is fairly simple therefore I do not think you will find a documentation.

Is there a way to produce the csv score file?

Use following to get csv prediction output.

java_command=java -Xmx6g -cp /usr/share/java/weka.jar 
$java_command weka.classifiers.Evaluation weka.classifiers.bayes.NaiveBayes -l irisNaiveBayes.model -T test.arff -classifications weka.classifiers.evaluation.output.prediction.CSV

We use Evaluation class instead of using Naive Bayes directly. First argument to Evaluation class is classifier to use, here NaiveBayes. -classification switch allows to output CSV.

Another example with well known iris data set.

java_command=java -Xmx6g -cp /usr/share/java/weka.jar 
$java_command weka.classifiers.bayes.NaiveBayes -t iris.arff  -d irisNaiveBayes.model

$java_command weka.classifiers.Evaluation weka.classifiers.bayes.NaiveBayes -l irisNaiveBayes.model -T iris.arff -classifications weka.classifiers.evaluation.output.prediction.CSV

This command gives following output.

=== Predictions on test data ===

inst#,actual,predicted,error,prediction
1,1:Iris-setosa,1:Iris-setosa,,1
2,1:Iris-setosa,1:Iris-setosa,,1
3,1:Iris-setosa,1:Iris-setosa,,1
4,1:Iris-setosa,1:Iris-setosa,,1
5,1:Iris-setosa,1:Iris-setosa,,1
6,1:Iris-setosa,1:Iris-setosa,,1
7,1:Iris-setosa,1:Iris-setosa,,1

Solved – Would a sorted class harm a 10-split cross validation

tl;dr: You're calling the wrong function! trainCV doesn't randomly partition the data.

Background

Weka has a few different ways to set up cross validation.

The "top-level" function is weka.classifiers.evaluation.Evalution. This class operates like (and presumably implements) the Experimenter GUI. If you do NOT provide a test set (and don't set the "no-cv" option), it will perform a stratified cross validation. The instance order is shuffled (and you can provide a seed for it). This class will take you a set of instances right through to a performance measurement (e.g., accuracy), which would obviate most of the code in your link. Use this class if you can.

Weka also provides "filters" that process sets of instances. There are two cross-fold validation-related filters: the supervised weka.filters.supervised.instance.StratifiedRemoveFolds and the unsupervised version weka.filters.unsupervised.instance.RemovedFolds. These shuffle your data and then create/remove the specified folds. You can provide a seed so that the fold assignments are reproducible across runs, if you like.

Finally, Instance also includes the weka.instance.trainCV/testCV pair of functions. These both come in two flavours. There is a three argument version, which takes a Java Random object and uses it to shuffle the data, and a two argument version which just blindly assigns the first $k/n$ points to the first fold (etc), without shuffling. This is potentially bad, as you've just discovered.

Your Problem

Your code uses the non-shuffling version of weka.instance.trainCV (see the code for CrossValidationSplit in "Step 3" of your code). Since the data are sorted by class, each fold is very unbalanced (even more than your whole data set), which is why your performance was initially terrible and it improves when you shuffle the data.

For best performance, I'd recommend trying a stratified cross validation with shuffling. If you can use the Evalution class, you can get the whole thing done in a handful of lines. Otherwise, check out the StratifiedRemoveFolds function linked above.

Source: the source (linked above)