Solved – What are the internal differences between classification and regression neural networks

classificationregressiontensorflow

I am quite new to neural networks, so please forgive any stupidity I express. This question asks why the rounded output of a regression model is not similar to the output of a classification model and also asks how a measure of confidence in outputs can be extracted from a classification model.

I am using skflow and TensorFlow to test regression and classification models on a simple dataset provided by sklearn (the iris dataset). I am creating the models by specifying a similar architecture for each of them. When I train the models on the dataset, I note that the classifier seems to massively outperform the regressor in the way that I have specified things.

Why is the performance so different? How wrong am I to try to convert the regressor to a classifier by rounding its output? How could a classifier express its individual outputs together with a measure of its confidence in these individual outputs?

My code is as follows:

#!/usr/bin/env python

from __future__ import division

import random
import numpy
import sklearn
import sklearn.datasets
import sklearn.metrics
import skflow

def main():

    epochs = 10000

    random.seed(42)

    dataset = sklearn.datasets.load_iris()

    # Create regression and classification models.

    model_regression = skflow.TensorFlowDNNRegressor(
        hidden_units  = [200, 300, 300, 300, 200],
        n_classes     = 0,
        learning_rate = 0.1,
        steps         = epochs
    )

    model_classification = skflow.TensorFlowDNNClassifier(
        hidden_units  = [200, 300, 300, 300, 200],
        n_classes     = 3,
        learning_rate = 0.1,
        steps         = epochs
    )

    # Train.

    model_regression.fit(dataset.data, dataset.target)
    model_classification.fit(dataset.data, dataset.target)

    # Print a listing of the target, the regression prediction rounded and the
    # classification prediction.

    print(
        "target",
        "regression prediction rounded",
        "classification prediction"
    )
    for target, prediction_regression, prediction_classification in zip(
        dataset.target,
        list(model_regression.predict(dataset.data)),
        list(model_classification.predict(dataset.data))
    ):
        print(
            target,
            int(round(float(prediction_regression[0]))),
            prediction_classification
        )

    # Calculate the prediction accuracies for the regression rounded and the
    # classification.

    regression_predictions_rounded = numpy.array([
        round(value) for value in model_regression.predict(dataset.data)
    ])
    score_regression = sklearn.metrics.accuracy_score(
        regression_predictions_rounded,
        dataset.target
    )

    score_classification = sklearn.metrics.accuracy_score(
        model_classification.predict(dataset.data),
        dataset.target
    )

    print("regression prediction accuracy on training dataset: {percentage}".format(
        percentage = 100 * score_regression
    ))

    print("classification prediction accuracy on training dataset: {percentage}".format(
        percentage = 100 * score_classification
    ))

if __name__ == "__main__":
    main()

Best Answer

My answer might not be the full answer/truth you are looking for but 8 month after asking I feel confident that it's better than nothing ^^

General difference Regression/Classification:

  • Classification means predicting a discrete valued output (e.g. type 1 or 2 or 3)
  • Regression means predicting a continuous-valued output (e.g. housing prices related to house size)

If you take a look at the sklearn examples here you will realize that the iris data is used for Linear Classifier, Neural Network, and a Custom Model (also a DNN). All those have in common that they are for classification. Linear Regression on this page uses boston.data instead.

What I want to say is that there are algorithms that better fit your data and those that are worse. iris data seems to be meant to be classified (they are for flowers cite Wikipedia "this data set became a typical test case for many statistical classification") and boston.data seems to be continuous-valued via regression (boston house-prices etc., classic for regression).

I can't tell you to what degree it is wrong to convert the regressor to a classifier but it's using the wrong math for the problem. It might work but sure has a downside like maybe you need a lot more training data. I'm not that deep into the math, sorry. But what I can say is even if one algorithm outperforms the other on your machine it doesn't mean it would be faster if you ran it on your GPU instead of CPU or in the Cloud.

Either you have a performance issue because of massive datasets and/or hardware limitations and need an algorithm that might not take care of everything or you take what is allegedly the best and test and compare with alternatives. Often e.g. with large datasets you can just drop some random datasets and solve your performance issue that way.

Related Question