MATLAB: Am I getting an error saying the Datastore is not shuffable

datastoredeepimagelearningprocessing

Why am I getting an error saying my Datastore is not shuffable?

Warning: Input datastore is not shuffleable but trainingOptions specified shuffling. Training will proceed without shuffling.
Error using nnet.internal.cnn.GeneralDatastoreDispatcher.splitDatastore (line 501)
The input datastore is not Partitionable and does not support parallel operations.

Best Answer

If you specify the "ExecutionEnvironment" in "trainingOptions" for a deep network to be "multiple-gpu" or another form of parallel processing, then it requires the underlying datastore to be partitionable. Most datastores are partitionable, but if you use the "combine" function, this may result in a new datastore that is not partitionable. There are two "solutions" to running into this error:

Specify a different execution environment that does not require a partitionable datastore.
Create a custom partitionable datastore. If you need to create a custom datastore that supports parallel or multi-GPU training, then your datastore must implement the "matlab.io.datastore.Partitionable" class.

You can check if a datastore is partitionable using the function "ispartitionable".

For more information on using parallel computing with datastores, please also refer to the following page:

https://www.mathworks.com/help/deeplearning/ug/datastores-for-deep-learning.html#mw_4e35721b-f50a-412a-af91-c35e606f52cc

Related Solutions

MATLAB: How can we use splitEachLabel for augmented image datastores

splitEachLabel function splits ImageDatastore labels by proportion. As you have augumented the image data store into augmentedImageDatastore, you can try partitioning it using partitionByIndex.

Suppose there are 5 observations in auds.

To assign 3 observations in auds1 use vector [1:3]

To assign rest 2 observations in auds2 use vector [4:5]

imds=imageDatastore(pathToDirectory,'LabelSource', 'foldernames', 'FileExtensions', {'.gif'});
auds = augmentedImageDatastore([227 227], imds);
auds1 = partitionByIndex(auds,[1:3]);
auds2 = partitionByIndex(auds,[4:5]);

For more information, see:

https://in.mathworks.com/help/deeplearning/ref/augmentedimagedatastore.partitionbyindex.html#d120e63199

MATLAB: Cannot utilize fully all GPUs during network training

Ok, problem solved.

as Suggested by Joss and Mathworks, MathWorks Support created a custom Datastore inheriting the properties involved in this procedure,

The Labels are a numeric vector as an input to the DS function but could but could be loaded from a .mat file as well.

for Training, apprently I needed to use the 'parallel' execution environment and define DispatchInBackground training option as 'true' (probably since I use AWS cloud service).

classdef matFilesDatastore < matlab.io.Datastore & ...
        matlab.io.datastore.Shuffleable & ...
        matlab.io.datastore.Partitionable
    
    properties
        Datastore
        Labels
        ReadSize
    end
    
    properties(SetAccess = protected)
        NumObservations
    end
    
    properties(Access = private)
        CurrentFileIndex
    end
    
    methods
        function ds = matFilesDatastore(folder, labels)
            % ds = matFilesDatastore(folder, labels) creates a datastore
            % from the data in folder and labels
            
            % Create file datastore
            fds = fileDatastore(folder, ...
                'ReadFcn',@readData, ...
                'IncludeSubfolders',true);
            ds.Datastore = fds;
            
            numObservations = numel(fds.Files);
            
            % Labels.
            ds.Labels = labels;
            
            % Initialize datastore properties.
            ds.ReadSize = 1;
            ds.NumObservations = numObservations;
            ds.CurrentFileIndex = 1;
        end
        
        function tf = hasdata(ds)
            % tf = hasdata(ds) returns true if more data is available.
            
            tf = ds.CurrentFileIndex + ds.ReadSize - 1 ...
                <= ds.NumObservations;
        end
        
        function [data,info] = read(ds)
            % [data,info] = read(ds) read one mini-batch of data.
            
            miniBatchSize = ds.ReadSize;
            info = struct;
            
            for i = 1:miniBatchSize
                predictors{i,1} = read(ds.Datastore);
                responses{i,1} = ds.Labels(ds.CurrentFileIndex);
                ds.CurrentFileIndex = ds.CurrentFileIndex + 1;
            end
            
            data = table(predictors,responses);
        end
        
        function reset(ds)
            % reset(ds) resets the datastore to the start of the data.          
            reset(ds.Datastore);
            ds.CurrentFileIndex = 1;
        end
        
        function dsNew = shuffle(ds)
            % dsNew = shuffle(ds) shuffles the files and the corresponding
            % labels in the datastore.
            
            % Create copy of datastore.
            dsNew = copy(ds);
            dsNew.Datastore = copy(ds.Datastore);
            fds = dsNew.Datastore;
            
            % Shuffle files and corresponding labels.
            numObservations = dsNew.NumObservations;
            idx = randperm(numObservations);
            fds.Files = fds.Files(idx);
            dsNew.Labels = dsNew.Labels(idx);
        end
        
        function subds = partition(ds, numPartitions, idx)
            subds = copy(ds);            
            subds.Datastore = partition(ds.Datastore, numPartitions, idx);
            subds.NumObservations = numel(subds.Datastore.Files);
            indices = pigeonHole(idx, numPartitions, ds.NumObservations);
            subds.Labels = ds.Labels(indices); 
            reset(subds);
        end
    end
    
    methods(Access = protected)
        function n = maxpartitions(ds)
            n = ds.NumObservations;
        end
    end
    
    methods (Hidden = true)
        function frac = progress(ds)
            % frac = progress(ds) returns the percentage of observations
            % read in the datastore.
            
            frac = (ds.CurrentFileIndex - 1) / ds.NumObservations;
        end
    end
end
function data = readData(filename)
% data = readData(filename) reads the data X from the MAT file
% filename
S = load(filename);
data = S.image;
% label = S.label;
end
function observationIndices = pigeonHole(partitionIndex, numPartitions, numObservations)
%pigeonHole   Helper function that maps partition index and numpartitions
%   to the corresponding observation indices.
    observationIndices = floor((0:numObservations - 1) * numPartitions / numObservations) + 1;    
    observationIndices = find(observationIndices == partitionIndex);
    % Convert to a vector if observationIndices is empty.
    if isempty(observationIndices)
        observationIndices = double.empty(0, 1);
    end
end

Best Answer

Related Solutions

MATLAB: How can we use splitEachLabel for augmented image datastores

MATLAB: Cannot utilize fully all GPUs during network training

Related Question