Let me suggest one scenario (the only one I can think of) where it might be useful to distinguish between "units" (or some similarly generic term) and "neurons." Biologically, a neuron is easy to identify, because it represents a single cell. In terms of neural nets, a neuron or "unit" has typically represented a single object, usually with one activation value, plus an additional threshold or separate input and output values in some cases. Problems arise in distinguishing between a neuron and a "unit" when we take into account the fact that the inputs, outputs, activations and thresholds of biological neurons are often mediated by multiple neurotransmitters and specific subsets of connections on the dendrites - many of which can be modeled as separate units. Then the line between "neuron" and "unit" blurs quickly. As William F. Allman puts it in pp. 65-66, Apprentices of Wonder: Inside the Neural Network Revolution (1989, Bantam Books: New York):
"An axon may release various amounts of transmitter; a receiving
dendrite might have varying amounts of receptor; the transmitter
itself may have different checmical properties and react at different
rates. And the whole process may be mitigated by the action of various
enzymes.”
Here's a more thorough treatment from Daniel Gardner (1993, The Neurobiology of Neural Networks, MIT Press: Cambridge, Mass.) (I lost the page number to this, so I can't provide an exact citation):
" First, it has become evident that neurons (both in vertebrates and
invertebrates) possess rich and complex intrinsic properties. Most
neurons have multiple channels to different ionic species, and these
channels can be regulated in a wide variety of ways: They can be
turned on or off by voltage, molecules, or ions. Some of these
channels can be active in the absence of external inputs to the cell,
and endow it with a variety of dynamic properties, such as the ability
to oscillate (Llinas 1988; Selverston 1988; Yamada et al. 1989). Thus,
it is not enough to specify the inputs to a neuron to predict its
outputs; its internal state will also determine its behavior. As a
consequence, neurons may be better represented as nonlinear dynamic
systems in their own right. For example, the intrinsic conductances of
thalamic neurons can allow them to act as linear input/output devices,
relaying information directly to cortex, but when they are
hyperpolarized, these conductances cause the neurons to burst,
significantly transforming their inputs (Llinas 1988). In terms of the
model neurons that have often been used in artificial neural networks,
the input/output relationship would need to be represented as a
function both of voltage and of time. "Second, the interactions
between neurons are complex. The differential distribution of synapses
on complex dendritic trees of neurons can significantly affect the
nature and intensity of their inputs to a neuron. In addition,
synapses may have multiple time courses (e.g., initial excitation,
slower inhibition, and still slower excitation [Getting and Dekin
1985), and connections may be dynamically reconfigured (e.g., by
inhibition of specific neurons [Getting and Dekin 1985, or by the
actions of neuromodulators [Harris-Warrick and Marder 1991; Marder and
Hooper 19851). Receptors controlling the synaptic response may be
gated both by the presence of a chemical, such as a neurotransmitter,
and by voltage, so that the synaptic connections between neurons can
be affected by their own activity and by the activity of neurons
impinging on them (discussions of these and other complexities in
synaptic interactions are found in chapters 2, 3, and 4). Influences
may occur over a variety of spatial and temporal scales: A
neuromodulator which is only slowly broken down may affect a very
large number of neurons in its vicinity over an appreciable period of
time as it diffuses away from its point of release. Furthermore, such
compounds are likely to selectively activate those subgroups of
neurons that have a receptor for that substance. Neuromodulators may
also have subtle but profound effects on the intrinsic properties of
neurons, activating or inactivating intrinsic currents and thus
changing their "electrical personality." Field potentials may alter
the excitability of neurons in different regions of the brain (Nunez
1981)."
I've run across other such quotes in the literature with similar detail, but those two should get the point across (Gardner's book may be a good starting point if you want to look into the matter further). In cases where we're dealing with multiple activations, thresholds and the like, it might be helpful to make a distinction between "neurons" and constituent "units" that contribute their own activations and other calculations; there's such bewildering complexity to these matters that I don't think anyone can give a definitive answer as to the best way to model such distinctions. I ran into this problem when trying to implement Fukushima's neocognitrons, in which each neuron has its own separate inhibitory and stimulatory inputs; first I tried modeling them as separate neurons, then as a single neuron with multiple outputs, but I'm still not certain what the optimal choice is. There may be solid computational advantages to modeling many of these various enzymes, neurotransmitters and receptors beyond mere biological plausibility; perhaps there's not; the whole topic is still far afield, even for neuroscientists, who still have much to learn about the purposes of such connections. I suspect such questions will become far more complex and pressing in the future once the field of neuroscience advances, enabling neural net researchers to mimic more of these internal calculations. For the time being it's safe to equate neurons with "units," but that might not be the case once more sophisticated neural nets begin to make practical use of this dizzying array of computations.
Let's start with a triviliaty: Deep neural network is simply a feedforward network with many hidden layers.
This is more or less all there is to say about the definition. Neural networks can be recurrent or feedforward; feedforward ones do not have any loops in their graph and can be organized in layers. If there are "many" layers, then we say that the network is deep.
How many layers does a network have to have in order to qualify as deep? There is no definite answer to this (it's a bit like asking how many grains make a heap), but usually having two or more hidden layers counts as deep. In contrast, a network with only a single hidden layer is conventionally called "shallow". I suspect that there will be some inflation going on here, and in ten years people might think that anything with less than, say, ten layers is shallow and suitable only for kindergarten exercises. Informally, "deep" suggests that the network is tough to handle.
Here is an illustration, adapted from here:
But the real question you are asking is, of course, Why would having many layers be beneficial?
I think that the somewhat astonishing answer is that nobody really knows. There are some common explanations that I will briefly review below, but none of them has been convincingly demonstrated to be true, and one cannot even be sure that having many layers is really beneficial.
I say that this is astonishing, because deep learning is massively popular, is breaking all the records (from image recognition, to playing Go, to automatic translation, etc.) every year, is getting used by the industry, etc. etc. And we are still not quite sure why it works so well.
I base my discussion on the Deep Learning book by Goodfellow, Bengio, and Courville which went out in 2017 and is widely considered to be the book on deep learning. (It's freely available online.) The relevant section is 6.4.1 Universal Approximation Properties and Depth.
You wrote that
10 years ago in class I learned that having several layers or one layer (not counting the input and output layers) was equivalent in terms of the functions a neural network is able to represent [...]
You must be referring to the so called Universal approximation theorem, proved by Cybenko in 1989 and generalized by various people in the 1990s. It basically says that a shallow neural network (with 1 hidden layer) can approximate any function, i.e. can in principle learn anything. This is true for various nonlinear activation functions, including rectified linear units that most neural networks are using today (the textbook references Leshno et al. 1993 for this result).
If so, then why is everybody using deep nets?
Well, a naive answer is that because they work better. Here is a figure from the Deep Learning book showing that it helps to have more layers in one particular task, but the same phenomenon is often observed across various tasks and domains:
We know that a shallow network could perform as good as the deeper ones. But it does not; and they usually do not. The question is --- why? Possible answers:
- Maybe a shallow network would need more neurons then the deep one?
- Maybe a shallow network is more difficult to train with our current algorithms (e.g. it has more nasty local minima, or the convergence rate is slower, or whatever)?
- Maybe a shallow architecture does not fit to the kind of problems we are usually trying to solve (e.g. object recognition is a quintessential "deep", hierarchical process)?
- Something else?
The Deep Learning book argues for bullet points #1 and #3. First, it argues that the number of units in a shallow network grows exponentially with task complexity. So in order to be useful a shallow network might need to be very big; possibly much bigger than a deep network. This is based on a number of papers proving that shallow networks would in some cases need exponentially many neurons; but whether e.g. MNIST classification or Go playing are such cases is not really clear. Second, the book says this:
Choosing a deep model encodes a very general belief that the function we
want to learn should involve composition of several simpler functions. This can be
interpreted from a representation learning point of view as saying that we believe
the learning problem consists of discovering a set of underlying factors of variation
that can in turn be described in terms of other, simpler underlying factors of
variation.
I think the current "consensus" is that it's a combination of bullet points #1 and #3: for real-world tasks deep architecture are often beneficial and shallow architecture would be inefficient and require a lot more neurons for the same performance.
But it's far from proven. Consider e.g. Zagoruyko and Komodakis, 2016, Wide Residual Networks. Residual networks with 150+ layers appeared in 2015 and won various image recognition contests. This was a big success and looked like a compelling argument in favour of deepness; here is one figure from a presentation by the first author on the residual network paper (note that the time confusingly goes to the left here):
But the paper linked above shows that a "wide" residual network with "only" 16 layers can outperform "deep" ones with 150+ layers. If this is true, then the whole point of the above figure breaks down.
Or consider Ba and Caruana, 2014, Do Deep Nets Really Need to be Deep?:
In this paper we provide empirical evidence that shallow nets are capable of learning the same
function as deep nets, and in some cases with the same number of parameters as the deep nets. We
do this by first training a state-of-the-art deep model, and then training a shallow model to mimic the
deep model. The mimic model is trained using the model compression scheme described in the next
section. Remarkably, with model compression we are able to train shallow nets to be as accurate
as some deep models, even though we are not able to train these shallow nets to be as accurate as
the deep nets when the shallow nets are trained directly on the original labeled training data. If a
shallow net with the same number of parameters as a deep net can learn to mimic a deep net with
high fidelity, then it is clear that the function learned by that deep net does not really have to be deep.
If true, this would mean that the correct explanation is rather my bullet #2, and not #1 or #3.
As I said --- nobody really knows for sure yet.
Concluding remarks
The amount of progress achieved in the deep learning over the last ~10 years is truly amazing, but most of this progress was achieved by trial and error, and we still lack very basic understanding about what exactly makes deep nets to work so well. Even the list of things that people consider to be crucial for setting up an effective deep network seems to change every couple of years.
The deep learning renaissance started in 2006 when Geoffrey Hinton (who had been working on neural networks for 20+ years without much interest from anybody) published a couple of breakthrough papers offering an effective way to train deep networks (Science paper, Neural computation paper). The trick was to use unsupervised pre-training before starting the gradient descent. These papers revolutionized the field, and for a couple of years people thought that unsupervised pre-training was the key.
Then in 2010 Martens showed that deep neural networks can be trained with second-order methods (so called Hessian-free methods) and can outperform networks trained with pre-training: Deep learning via Hessian-free optimization. Then in 2013 Sutskever et al. showed that stochastic gradient descent with some very clever tricks can outperform Hessian-free methods: On the importance of initialization and momentum in deep learning. Also, around 2010 people realized that using rectified linear units instead of sigmoid units makes a huge difference for gradient descent. Dropout appeared in 2014. Residual networks appeared in 2015. People keep coming up with more and more effective ways to train deep networks and what seemed like a key insight 10 years ago is often considered a nuisance today. All of that is largely driven by trial and error and there is little understanding of what makes some things work so well and some other things not. Training deep networks is like a big bag of tricks. Successful tricks are usually rationalized post factum.
We don't even know why deep networks reach a performance plateau; just 10 years people used to blame local minima, but the current thinking is that this is not the point (when the perfomance plateaus, the gradients tend to stay large). This is such a basic question about deep networks, and we don't even know this.
Update: This is more or less the subject of Ali Rahimi's NIPS 2017 talk on machine learning as alchemy: https://www.youtube.com/watch?v=Qi1Yry33TQE.
[This answer was entirely re-written in April 2017, so some of the comments below do not apply anymore.]
Best Answer
In theory, yes. In practice, things are more subtle.
First of all, let's clear the field from a doubt raised in the comments: neural networks can handle multiple outputs in a seamless fashion, so it doesn't really matter whether we consider multiple regression or not (see The Elements of Statistical Learning, paragraph 11.4).
Having said that, a neural network of fixed architecture and loss function would indeed just be a parametric nonlinear regression model. So it would even less flexible than nonparametric models such as Gaussian Processes. To be precise, a single hidden layer neural network with a sigmoid or tanh activation function would be less flexible than a Gaussian Process: see http://mlss.tuebingen.mpg.de/2015/slides/ghahramani/gp-neural-nets15.pdf. For deep networks this is not true, but it becomes true again when you consider Deep Gaussian Processes.
So, why are Deep Neural Networks such a big deal? For very good reasons:
They allow fitting models of a complexity that you wouldn't even begin to dream of, when you fit Nonlinear Least Squares models with the Levenberg-Marquard algorithm. See for example https://arxiv.org/pdf/1611.05431.pdf, https://arxiv.org/pdf/1706.02677.pdf and https://arxiv.org/pdf/1805.00932.pdf where the number of parameters $p$ goes from 25 to 829 millions. Of course DNNs are overparametrized, non-identifiable, etc. etc. so the number of parameters is very different from the "degrees of freedom" of the model (see https://arxiv.org/abs/1804.08838 for some intuition). Still, it's undeniably amazing that models with $N <<p$ ($N=$ sample size) are able to generalize so well.
They scale to huge data sets. A vanilla Gaussian Process is a very flexible model, but inference has a $O(N^3)$ cost which is completely unacceptable for data sets as big as ImageNet or bigger such as Open Image V4. There are approximations to inference with GPs which scale as well as NNs, but I don't know why they don't enjoy the same fame (well, I have my ideas about that, but let's not digress).
For some tasks, they're impressively accurate, much better than many other statistical learning models. You can try to match ResNeXt accuracy on ImageNet, with a 65536 inputs kernel SVM, or with a random forest for classification. Good luck with that.
However, the real difference between theory:
and practice in my opinion, is that in practice nothing about a deep neural network is really fixed in advance, so you end up fitting a model from a much bigger class than you would expect. In real-world applications, none of these aspects are really fixed:
Look how many choices are performed even in a relatively simple case where there is a strong seasonal signal, and the number of features is small, as far as DNNs go:
https://stackoverflow.com/questions/48929272/non-linear-multivariate-time-series-response-prediction-using-rnn
Thus in practice, even though ideally fitting a DNN would mean to just fit a model of the type
$y=f(\mathbf{x}\vert\boldsymbol{\theta})+\epsilon$
where $f$ has a certain hierarchical structure, in practice very little (if anything at all) about the function and the fitting method is defined in advance, and thus the model is much more flexible than a "classic" parametric nonlinear model.