MATLAB: How to check the robustness of the Neural network model

classificationmachine learningneural network

Hi everyone,
I need your help for my project.
  • I used Neural Pattern Recognition app, and have already built an NN model for classification with 4 labels. The NN model worked very well. Accuracy classification reaches more than 90%.
  • However, when I want to check this model with new data (new data = the original data through an AWGN channel having a 10 dB signal-to-noise ratio (SNR). The classification result is always less than 30% accuracy.
  • How to create an augmented dataset and train it?
  • How to check the robustness of the Neural network model?
Please help me!!!
Code to create a neural network:
% Solve a Pattern Recognition Problem with a Neural Network
% Script generated by Neural Pattern Recognition app
%% Load data
%   X - input data.
%   Y - target data.
load('mydata.mat')
x = X;
t = Y;
%% Choose a Training Function
trainFcn = 'trainscg';  % Scaled conjugate gradient backpropagation.
%% Create a Pattern Recognition Network
hiddenLayerSize = 50;
net = patternnet(hiddenLayerSize, trainFcn);
% Choose Input and Output Pre/Post-Processing Functions
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};
% Setup Division of Data for Training, Validation, Testing
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
% Choose a Performance Function
net.performFcn = 'crossentropy';  % Cross-Entropy
% Choose Plot Functions
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
    'plotconfusion', 'plotroc'};
%% Train the Network
[net,tr] = train(net,x,t);
%% Test the Network

y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
%% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)
%% View the Network and plots confusion matrix
view(net)
figure, plotconfusion(t,y)
save ('mynet');
Code to check neural network with data added white noise:
clc;
clear all;
%load
load('mynet');
%% add Gaussian white noise SNR=10dB
load('mydata.mat')
X10dB = awgn(X,10,'measured');
x = X10dB; % train data
t = Y; % target
%% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)
%% Plots
figure, plotconfusion(t,y)

Best Answer

If you are going to test with white noise, include white noise in your design (i.e., training + validation)
Then, given a fixed input level of white noise for design (i.e., design + noise1) you can obtain individual performance measures of training, validation and test as a function of added noise level.
Hope this helps,
Thank you for formally accepting my answer
Greg
Related Question