Solved – Poor accuracy with a keras neural network

kerasneural networksword embeddings

I'm trying to understand why my NN doesn't predict at all.
I am using an embedding layer from gensim into keras to make a binary classification of paragraphs of text (similar to twitter sentiment analysis).

Here's the code :

from keras.models import Sequential
from keras.layers import Flatten
from keras.layers import Dense

keras_model = Sequential()
keras_model.add(embedding) // Added the embedding layer from gensim
keras_model.add(Flatten())
keras_model.add(Dense(1, activation='softmax'))
keras_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])

I'll try to describe in more details my attempts so far :

  1. Initially I've trained the model using a dataset consisting of ~220k samples and I had 92.85% accuracy, which was great , but then I noticed that the ratio between negative and positive samples was exactly 0.928, which meant I needed to clean my dataset.

2 .I made the dataset with 50/50 distribution of positive to negative samples (~26k samples) then I tried the same and got accuracy of 50%.

3.Played around with different activations (relu, softmax , sigmoid) – no change or it dropped to 0% accuracy.

4.Added an extra hidden layer – again no change.

5.Tried different batch sizes (6,32,128,1024) – no change.

keras_model.summary()

Layer (type)                 Output Shape              Param #   
=================================================================
embedding_2 (Embedding)      (None, 500, 100)          596200    
_________________________________________________________________
flatten_5 (Flatten)          (None, 50000)             0         
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 50001     
=================================================================
Total params: 646,201
Trainable params: 50,001
Non-trainable params: 596,200
_________________________________________________________________

Is there anything that can be done to get some real accuracy from this neural netowork ? Am I doing something wrong or the dataset is small to have a neural network as a classifier.

Best Answer

You apply softmax activation function on the output layer with only one output neuron. Softmax has to sum to 1 over all output neurons, and since you have only one of them, that one neuron will always output 1. Switch softmax to sigmoid and you're good.

Related Question