MATLAB: Multi-Input CNN for image classification

multi-input_cnn

I have two different folders of images for 5 objects. I want to create a multi-input CNN as attached here. How can I send two different folders in trainNetwork while training the network and for testing as well?

Best Answer

In order to train Multiple input networks you have to use a datastore & After referring to Input Arguments & ds of trainNetwork, Multiple-Input and Multiple-Output Networks, Input Datastore for Training, Validation, and Inference, Read One Array at a Time From Large MAT-File the following is one possible way:
Save all the categorical labels in a mat file & While loading the labels in command window it should look like below:(N(number of training samples) = 4000 in the below case)
>> labels = load('labelData.mat')
labels = 
  struct with fields:
    labels: [4000×1 categorical]
Note that the order of the labels should be i.e., sorted in the same order of the files in the imageDatastore used for input1 and input2.
trainpath1 = fullfile("PATH TO INPUT1 IMAGES");
trainpath2 = fullfile("PATH TO INPUT2 IMAGES");
imds1 = imageDatastore(trainpath1, 'IncludeSubfolders',true, 'FileExtensions','.PNG','LabelSource','foldernames');
imds2 = imageDatastore(trainpath2, 'IncludeSubfolders',true, 'FileExtensions','.PNG','LabelSource','foldernames');
labelds = fileDatastore('labelData.mat','ReadFcn',@myReadFcn,'ReadMode','partialfile');
cds = combine(imds1,imds2,labelds);
% Change the image sizes accordingly
imsize1 = [32 80 3];
imsize2 = [48 64 3];
numClasses = 5;
%% Define muliple input network
layers1 = [
    imageInputLayer(imsize1,'Name','input1')  
    convolution2dLayer(3,16,'Padding','same','Name','conv_1')
    reluLayer('Name','relu_1')
    fullyConnectedLayer(10,'Name','fc11')
    additionLayer(2,'Name','add')
    fullyConnectedLayer(numClasses,'Name','fc12')
    softmaxLayer('Name','softmax')
    classificationLayer('Name','classOutput')];
lgraph = layerGraph(layers1);
layers2 = [imageInputLayer(imsize2,'Name','input2')
    convolution2dLayer(3,16,'Padding','same','Name','conv_2')
    reluLayer('Name','relu_2')
    fullyConnectedLayer(10,'Name','fc21')];
lgraph = addLayers(lgraph,layers2);
lgraph = connectLayers(lgraph,'fc21','add/in2');
plot(lgraph)
%% Define trainingOptions and also set 'Shuffle' to 'never' for this workaround to work
options = trainingOptions('adam', ...
    'InitialLearnRate',0.005, ...
    'LearnRateSchedule','piecewise',...
    'MaxEpochs',3, ...
    'MiniBatchSize',1, ...
    'Verbose',1, ...
    'Plots','training-progress');
net = trainNetwork(cds,lgraph,options);
%% custom read function to be used for the labels fileDatastore 
%to ensure the read(fileDatastore) returns 1 label at a time
function [data,userdata,done] = myReadFcn(filename,userdata)
% If variable list is empty, 
% create list of variables from the file
if isempty(userdata) 
    labels = load(filename);
    userdata = labels.labels;
end
% Load a variable from the list of variables
data = userdata(1);
% Remove the newly-read variable from the list
userdata(1) = []; 
% Move on to the next file if this file is done reading.
done = isempty(userdata); 
end
For testing the network:
testimds1 = imageDatastore(testpath1);
testimds2 = imageDatastore(testpath2);
testcds = combine(testimds1,testimds2);
YPred = predict(net,testcds,'MiniBatchSize',4)
Related Question