MATLAB: Mnist classification using batch method

Hi. I want to train a neural network with mnist database using batch method. I use below code but my accuracy is very low. but I think the code is correct. can any one help me please?

function [hiddenWeights, outputWeights, error] = train_network_batch(numberOfHiddenUnits, input, target, epochs, batchSize, learningRate,lambda)

    % The number of training vectors.
    trainingSetSize = size(input, 2);
    % Input vector has 784 dimensions.
    inputDimensions = size(input, 1);
    % We have to distinguish 10 digits.
    outputDimensions = size(target, 1);
    % Initialize the weights for the hidden layer and the output layer.
    %     hiddenWeights = randn(NHiddenUnit, inputDimensions)*1/sqrt(size(input, 1));
%     outputWeights = randn(outputDimensions, NHiddenUnit)*1/sqrt(size(input, 1));
    hiddenWeights = rand(numberOfHiddenUnits, inputDimensions);
    outputWeights = rand(outputDimensions, numberOfHiddenUnits);
    hiddenWeights = hiddenWeights./size(hiddenWeights, 2);
    outputWeights = outputWeights./size(outputWeights, 2);
     hiddenWeights_store = hiddenWeights;
    outputWeights_store = outputWeights;
    n = zeros(batchSize,1);
    validation_count=0;
    validation_accuracy=0;
    figure; hold on;
 %batch method
     for t = 1: epochs
        for k = 1: batchSize
            % Select which input vector to train on.
            n(k) = floor(rand(1)*trainingSetSize + 1);
%             n(k) =k;
            % Propagate the input vector through the network.
            inputVector = input(:, n(k));
            hiddenActualInput = hiddenWeights*inputVector;
            hiddenOutputVector = linear_func(hiddenActualInput);
            outputActualInput = outputWeights*hiddenOutputVector;
            outputVector = linear_func(outputActualInput);
            targetVector = target(:, n(k));
            % Backpropagate the errors.
            outputDelta = dlinear_func(outputActualInput).*(outputVector - targetVector);
            hiddenDelta = dlinear_func(hiddenActualInput).*(outputWeights'*outputDelta);

% outputWeights_store = outputWeights_store -(learningRate*lambda/batchSize).*outputWeights- learningRate.*outputDelta*hiddenOutputVector'; hiddenWeights_store = hiddenWeights_store -(learningRate*lambda/batchSize).*hiddenWeights-learningRate.*hiddenDelta*inputVector';

% outputWeights =(1-(learningRate*lambda/batchSize)).*outputWeights – learningRate.*outputDelta*hiddenOutputVector'; % hiddenWeights = (1-(learningRate*lambda/batchSize)).*hiddenWeights – learningRate.*hiddenDelta*inputVector';

        end;
        outputWeights=outputWeights+(outputWeights_store./batchSize);
        hiddenWeights=hiddenWeights+(hiddenWeights_store./batchSize);
        outputWeights_store=0;
        hiddenWeights_store=0;

% %*********************************end of batch method*************** % Calculate the error for plotting. error = 0; for k = 1: batchSize inputVector = input(:, n(k)); targetVector = target(:, n(k));

            error = error + norm(linear_func(outputWeights*linear_func(hiddenWeights*inputVector)) - targetVector, 2);
        end;
        error = error/batchSize;
        plot(t, error,'*');
        title(['MSE_ batch','NH= ',num2str(numberOfHiddenUnits),',',' alfa=',num2str(learningRate),' ,epoch=',num2str(epochs)]);
        xlabel('epoch');
        ylabel('cost');
        inputValues=load('validation.mat');
        inputValues=inputValues.v;
        labels=load('label.mat');
        labels=labels.l;
        [correctlyClassified, classificationErrors]=validation_network(hiddenWeights,outputWeights,inputValues',labels);
         correctlyClassified=correctlyClassified/10000;
         if correctlyClassified<= validation_accuracy 
            validation_count=validation_count+1;
         else
             validation_count=0;
         end
         if validation_count>7
             break;
         end
         validation_accuracy=correctlyClassified;
    end;
end

Best Answer

1. I don't think that anyone wants to wade through all of that code when you can just use MATLAB classification functions

 help PATTERNNET
 doc PATTERNNET

2. If none of your hidden or output functions is nonlinear, then all you have is a complicated linear classifier which can be implemented with BACKSLASH.

Hope this helps.

Thank you for formally accepting my answer

Greg

Related Solutions

MATLAB: True Positive and False Positve rate of classification neural network

Yes.

MATLAB: Given feed back that, I need to call outputSummary with the proper arguments.

This code is so badly written. There are too many nested functions.

outputSummary function is defined inside the costFunction. theCostFunction is called inside the outputSummary.

You can not call outputSummary from outside because it is already nested inside the costFunction.

What is the purpose of this code what it is expected to do?

I just changed couple of things and moved your outputSummary to the outside.

I called outputSummary inside your for loop because it seems that this function uses the inputs which are defined inside the for loop iterations.

Check this:

clear all;
commandwindow;
%Using Matlab, create a multi-layer perceptron with 3 layers: input layer,
%hidden layer, output layer (using a sigmoid function).
%Define the learning rate and total iterations
learningRate = 0.5;
totalIterations = 500;
%Define the size of the input layer and the hidden layer:
inputLayerNumber = 2;
hiddenLayerNumber = 2;
%Define the input and hidden layer:
inputLayer = zeros(inputLayerNumber, 1);
hiddenLayer = zeros(hiddenLayerNumber, 1);
%Add the bias to the input and hidden layer:
inputLayerWithBias = zeros(inputLayerNumber + 1, 1);
hiddenLayerWithBias = zeros(hiddenLayerNumber + 1, 1);
%Define the output layer:
outputLayer = 0;
%Randomly assign the weights to the input and hidden layer:
inputLayerWeights = rand( (inputLayerNumber + 1) ,hiddenLayerNumber) - .5 ;
hiddenLayerWeights = rand( (hiddenLayerNumber + 1), 1) - .5;
%Define the input data:
inputLayer = [0 0; 0 1; 1 0; 1 1];
%Define the target output for the input layer:
ANDtargetOutput = [0; 0; 0; 1];
targetOutput = ANDtargetOutput;
%Define the variable 'm' as the number of samples:
m = size(targetOutput, 1);
inputLayerWithBias = [ones(m,1) inputLayer];
%Create a for loop, that will step through each of the samples one at a time
for iter=1:totalIterations
    for i = 1:m
        hiddenLayerActivation = inputLayerWithBias(i, :) * inputLayerWeights;
        hiddenLayer = sigmoid(hiddenLayerActivation);
        
        %Add the bias to the hiddenLayer
        hiddenLayerWithBias = [1, hiddenLayer];
        outputLayer = sigmoid(hiddenLayerWithBias * hiddenLayerWeights);
        
        %Calculate the error:
        deltaOutput = targetOutput(i) - outputLayer;
        deltaHidden(1) = (deltaOutput * hiddenLayerWeights(1)) .* ((hiddenLayerWithBias(1) * (1.0 - hiddenLayerWithBias(1))));
        deltaHidden(2) = (deltaOutput * hiddenLayerWeights(2)) .* ((hiddenLayerWithBias(2) * (1.0 - hiddenLayerWithBias(2))));
        deltaHidden(3) = (deltaOutput * hiddenLayerWeights(3)) .* ((hiddenLayerWithBias(3) * (1.0 - hiddenLayerWithBias(3))));
        % Fixed Step Gradient Descent - Update the weights
        hiddenLayerWeights(1) = hiddenLayerWeights(1) + (learningRate * (deltaOutput * hiddenLayerWithBias(1)));
        hiddenLayerWeights(2) = hiddenLayerWeights(2) + (learningRate * (deltaOutput * hiddenLayerWithBias(2)));
        hiddenLayerWeights(3) = hiddenLayerWeights(3) + (learningRate * (deltaOutput * hiddenLayerWithBias(3)));
        %update each weight according to the part that they played
        inputLayerWeights(1,1) = inputLayerWeights(1,1) + (learningRate * deltaHidden(2) * inputLayerWithBias(i, 1));
        inputLayerWeights(1,2) = inputLayerWeights(1,2) + (learningRate * deltaHidden(3) * inputLayerWithBias(i, 1));
        inputLayerWeights(2,1) = inputLayerWeights(2,1) + (learningRate * deltaHidden(2) * inputLayerWithBias(i, 2));
        inputLayerWeights(2,2) = inputLayerWeights(2,2) + (learningRate * deltaHidden(3) * inputLayerWithBias(i, 2));
        inputLayerWeights(3,1) = inputLayerWeights(3,1) + (learningRate * deltaHidden(2) * inputLayerWithBias(i, 3));
        inputLayerWeights(3,2) = inputLayerWeights(3,2) + (learningRate * deltaHidden(3) * inputLayerWithBias(i, 3));
        outputSummary(inputLayerWithBias, inputLayerWeights,hiddenLayerWeights, targetOutput, totalIterations)
    end
end
%Create a function that will summarize the output of the 4 samples:
function outputSummary(inputLayerWithBias, inputLayerWeights,hiddenLayerWeights, targetOutput, totalIterations)
cost = costFunction(inputLayerWithBias, inputLayerWeights,hiddenLayerWeights, targetOutput);
hiddenLayer = sigmoid(inputLayerWithBias * inputLayerWeights);
%we have multiple samples, so we need to add the bias to each of them
hiddenLayerWithBias = [ones(size(targetOutput,1),1) hiddenLayer];
actualOutput = sigmoid(hiddenLayerWithBias * hiddenLayerWeights);
fprintf('\n\n=========================================\n');
fprintf('Output Summary (after %d iterations):\n', totalIterations);
fprintf('Total Cost: [%f]\n', cost);
for i=1:length(actualOutput)
    if(actualOutput(i) > 0.5)
        thresholdedValue = 1;
    else
        thresholdedValue = 0;
    end
    if(thresholdedValue == targetOutput(i))
        fprintf('Sample [%d]: Target = [%f} Thresholded Value = [%f] Actual= [%f]\n',i, targetOutput(i), thresholdedValue, actualOutput(i));
    else % else print the error in red
        fprintf(2,'Sample[%d]: Target = [%f] Thresholded Value = [%f] Actual= [%f]\n', i, targetOutput(i), thresholdedValue, actualOutput(i));
    end
end
fprintf('=========================================\n\n\n');
end
%Create the sigmoid function:
function a = sigmoid(z)
a = 1.0 ./ (1.0 + exp(-z));
end
%Create the cost function:
% This function will only work for NN with just one output (k = 1)
function [averageCost] = costFunction(inputLayerWithBias,inputLayerWeights, hiddenLayerWeights, targetOutput)
%Sum of square errors cost function
m = 4;
hiddenLayer = sigmoid(inputLayerWithBias * inputLayerWeights);
hiddenLayerWithBias = [ones(m,1) hiddenLayer];
outputLayer = sigmoid(hiddenLayerWithBias * hiddenLayerWeights);
% Step through all of the samples and calculate the cost at each one
for i=1:m
    cost(i) = (1/2) * ((outputLayer(i) - targetOutput(i)) .^ 2);
end
%Sum up all of the individual costs
totalCost = sum(cost);
%average them out
averageCost = totalCost * (1/m);
%end
end

Best Answer

Related Solutions

MATLAB: True Positive and False Positve rate of classification neural network

MATLAB: Given feed back that, I need to call outputSummary with the proper arguments.

Related Question