Solved – Neuron vs. unit in a neural network

neural networksterminology

In the context of machine learning, is there any difference between the terms unit and neuron? I have read that some people prefer to use the term unit as neurons in an ANN have little in common with neurons in the human brain (see quote below). Is there any other difference?

Quote from Michael A. Nielsen, "Neural Networks and Deep Learning", Determination Press, 2015 (Creative Commons Attribution-NonCommercial 3.0 Unported License.)

The origins of convolutional neural networks go back to the 1970s. But the seminal paper establishing the modern subject of convolutional networks was a 1998 paper, "Gradient-based learning applied to document recognition", by Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. LeCun has since made an interesting remark on the terminology for convolutional nets: "The [biological] neural inspiration in models like convolutional nets is very tenuous. That's why I call them 'convolutional nets' not 'convolutional neural nets', and why we call the nodes 'units' and not 'neurons' ". Despite this remark, convolutional nets use many of the same ideas as the neural networks we've studied up to now: ideas such as backpropagation, gradient descent, regularization, non-linear activation functions, and so on. And so we will follow common practice, and consider them a type of neural network. I will use the terms "convolutional neural network" and "convolutional net(work)" interchangeably. I will also use the terms "[artificial] neuron" and "unit" interchangeably

Best Answer

Let me suggest one scenario (the only one I can think of) where it might be useful to distinguish between "units" (or some similarly generic term) and "neurons." Biologically, a neuron is easy to identify, because it represents a single cell. In terms of neural nets, a neuron or "unit" has typically represented a single object, usually with one activation value, plus an additional threshold or separate input and output values in some cases. Problems arise in distinguishing between a neuron and a "unit" when we take into account the fact that the inputs, outputs, activations and thresholds of biological neurons are often mediated by multiple neurotransmitters and specific subsets of connections on the dendrites - many of which can be modeled as separate units. Then the line between "neuron" and "unit" blurs quickly. As William F. Allman puts it in pp. 65-66, Apprentices of Wonder: Inside the Neural Network Revolution (1989, Bantam Books: New York):

"An axon may release various amounts of transmitter; a receiving dendrite might have varying amounts of receptor; the transmitter itself may have different checmical properties and react at different rates. And the whole process may be mitigated by the action of various enzymes.”

Here's a more thorough treatment from Daniel Gardner (1993, The Neurobiology of Neural Networks, MIT Press: Cambridge, Mass.) (I lost the page number to this, so I can't provide an exact citation):

" First, it has become evident that neurons (both in vertebrates and invertebrates) possess rich and complex intrinsic properties. Most neurons have multiple channels to different ionic species, and these channels can be regulated in a wide variety of ways: They can be turned on or off by voltage, molecules, or ions. Some of these channels can be active in the absence of external inputs to the cell, and endow it with a variety of dynamic properties, such as the ability to oscillate (Llinas 1988; Selverston 1988; Yamada et al. 1989). Thus, it is not enough to specify the inputs to a neuron to predict its outputs; its internal state will also determine its behavior. As a consequence, neurons may be better represented as nonlinear dynamic systems in their own right. For example, the intrinsic conductances of thalamic neurons can allow them to act as linear input/output devices, relaying information directly to cortex, but when they are hyperpolarized, these conductances cause the neurons to burst, significantly transforming their inputs (Llinas 1988). In terms of the model neurons that have often been used in artificial neural networks, the input/output relationship would need to be represented as a function both of voltage and of time. "Second, the interactions between neurons are complex. The differential distribution of synapses on complex dendritic trees of neurons can significantly affect the nature and intensity of their inputs to a neuron. In addition, synapses may have multiple time courses (e.g., initial excitation, slower inhibition, and still slower excitation [Getting and Dekin 1985), and connections may be dynamically reconfigured (e.g., by inhibition of specific neurons [Getting and Dekin 1985, or by the actions of neuromodulators [Harris-Warrick and Marder 1991; Marder and Hooper 19851). Receptors controlling the synaptic response may be gated both by the presence of a chemical, such as a neurotransmitter, and by voltage, so that the synaptic connections between neurons can be affected by their own activity and by the activity of neurons impinging on them (discussions of these and other complexities in synaptic interactions are found in chapters 2, 3, and 4). Influences may occur over a variety of spatial and temporal scales: A neuromodulator which is only slowly broken down may affect a very large number of neurons in its vicinity over an appreciable period of time as it diffuses away from its point of release. Furthermore, such compounds are likely to selectively activate those subgroups of neurons that have a receptor for that substance. Neuromodulators may also have subtle but profound effects on the intrinsic properties of neurons, activating or inactivating intrinsic currents and thus changing their "electrical personality." Field potentials may alter the excitability of neurons in different regions of the brain (Nunez 1981)."

I've run across other such quotes in the literature with similar detail, but those two should get the point across (Gardner's book may be a good starting point if you want to look into the matter further). In cases where we're dealing with multiple activations, thresholds and the like, it might be helpful to make a distinction between "neurons" and constituent "units" that contribute their own activations and other calculations; there's such bewildering complexity to these matters that I don't think anyone can give a definitive answer as to the best way to model such distinctions. I ran into this problem when trying to implement Fukushima's neocognitrons, in which each neuron has its own separate inhibitory and stimulatory inputs; first I tried modeling them as separate neurons, then as a single neuron with multiple outputs, but I'm still not certain what the optimal choice is. There may be solid computational advantages to modeling many of these various enzymes, neurotransmitters and receptors beyond mere biological plausibility; perhaps there's not; the whole topic is still far afield, even for neuroscientists, who still have much to learn about the purposes of such connections. I suspect such questions will become far more complex and pressing in the future once the field of neuroscience advances, enabling neural net researchers to mimic more of these internal calculations. For the time being it's safe to equate neurons with "units," but that might not be the case once more sophisticated neural nets begin to make practical use of this dizzying array of computations.

Related Question