Solved – Difference between linear regression and neural network

linear modelmachine learningneural networksregression

I am obviously confused with terms, and different concepts behind it. Each websites gives different intuitions. With all intuitions my brain is full of confusion now. Please help me to address what is right.

  1. Neural Network = Multi Layer Perceptron
  2. Linear Network/Regression = Neural Network ( with No hidden layer) only input and output layer.

This Link proves linear regression without hidden layer.

Now the confusion is with respect to binary output and continuous value?

Can I claim below points are also called Linear Regression?

  1. The neural network with binary output with one or more hidden layers.

  2. Some site claims linear regression means the continuous value output. If I have an MLP with hidden layers, and its output is continuous value (ex: house price), then is it called linear regression?

  3. Neural Network with linear activation functions ( doesn't matter binary output, continuous output value, hidden layer)

Hope you understood my confusion.

Best Answer

Linear regression is defined in terms of a linear function:

$$ \hat y = w_1 x_1 + w_2 x_2 + \dots + w_k x_k + b $$

where $\hat y$ is the prediction for the target variable $y$, $x_1,x_2,\dots,x_k$ are the features, $w_1,w_2,\dots,w_k$ are the weights and $b$ is a bias term.

In terms of neural networks, you have multiple inputs $x_1,x_2,\dots,x_k$ and a single output $\hat y$. So there is a single densely-connected layer, a single output layer with one unit and no hidden layer. There is also no activation (or link function), as with the link function we would rather be talking about generalized linear models (e.g. logistic regression). But there is more in the definition of GLMs than just the link function. Linear regression means also using squared loss and no regularization.

Using linear regression for predicting binary outputs is a suboptimal choice, same for counts, and there are specialized GLMs for many different problems. So basically yes, we define and use linear regression for continuous outputs.

All this said, I don't really think that calling linear regression a neural network makes much sense. Sure, there are people who will call it a neural network or even artificial intelligence, but unless you want to monetize your startup and need to make much marketing fuss around the big data blockchain deep learning artificial intelligence that you do, it would sound rather ridiculous.

Related Question