I have a doubt regarding an exercise here. Suppose that we have a neural network that tries to map a $28\times 28$ image of a digit to what digit it actually represents. So we have a neural network like this:
The activation function is calculated for each output neuron and the neuron with the highest activation value ends up "firing" (i.e. if output neuron 1 has the highest activation function value, then the model thinks the digit is 1).
With that background, here's the actual problem:
There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first $3$ layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least $0.99$, and incorrect outputs have activation less than $0.01$.
So here I'm assuming the "perceptron rule", i.e. the following activation function for the final layer:
$y^{(i)} = 0$ if $(b + \sum_jw^{(i)}_jx_j) \leq 0$
$y^{(i)} = 1$ if $(b + \sum_jw^{(i)}_jx_j) > 0$
where $j \in \{0,1,2,\ldots,9\}$ and $i \in \{1,2,3,4\}$. Since $x_i < 0.01$ for any incorrect output, I guess the weight $w^{(i)}_j=1$ if the $i$-th bit in the binary representation of digit $j$ is $1$. As an example, since $9$ is $1001$ in binary, $w^{(1)}_9=w^{(4)}_9=1$ and $w^{(2)}_9=w^{(3)}_9=0$.
In that scenario, the maximum number of digits that have $1$ in any of the $4$ bits is $5$ – that's because the $4$-th bit of $1,3,5,7,9$ is $1$. A worst case scenario is if the actual digit is $6$, so $x_6 \geq 0.99$ and all other $x_i$'s $<0.01$. Also, $w^{(1)}_6 =w^{(3)}_6= w^{(5)}_6=w^{(7)}_6=w^{(9)}_6=1$, which means $\sum_jw^{(i)}_jx_j$ is only slightly less than $0.05$, meaning that the bias term $b=-0.05$.
Is this approach/answer correct? I'm not sure if I'm missing something here. Thanks in advance!
Best Answer
Your reasoning, for the most part, is correct. However, as the sigmoid function deals with real numbers in the interval $[0, 1]$, the weights and biases you've chosen are not ideal. Suppose the input was an image of the number $3$, in this case, the $3$rd ($4$th if you're counting from 1) neuron in the third layer is $> 0.99$ while all the other neurons in that layer are $< 0.01$.
Here, $w.x+b$ according to your set of weights and biases give a result of some number $ > 0.94$ for the $0$th output neuron. Taking $z = 0.94$ as input to the sigmoid function, we'd get an output of around $0.28$ for the $0$th neuron. This would logically be a 0. But, for a 3, this neuron must output a 1.
This is happening as the inputs are bounded in the interval $[0, 1]$. If the weights are only $0$ or $1$, the effective value of $z=w.x+b$ can atmost be a one and in the worst case, b. As the term $e^{-z}$ requires a bigger value for $z$ to make it zero, consider increasing the weights to be $10$ and $-10$ instead of $1$ and $0$. The bias can be any value as it doesn't play a major role in deciding the output of a sigmoid neuron, atleast in this particular case.