MATLAB: Error in loading data from AWS S3

awsdata storageMATLABs3

I tried to get a files from a bucket i creat in S3

I use a code i saw in different places:

setenv('AWS_ACCESS_KEY_ID', <myKeyID>); 
setenv('AWS_SECRET_ACCESS_KEY', <myaccessKey>); 
setenv('AWS_REGION', 'us-east-1');
fds = fileDatastore('s3://{mybucket}/','ReadFcn',@load, 'FileExtensions',{'.csv'})

matlab "think" a lot and then I got

Error using fileDatastore (line 102)
Cannot find files or folders matching: 's3://{mybucket}/'.

I try to change to a bucket that don't exist and i got the same error but vary fast, I tried to change the ReadFcn, I try to get directly one file (and not all the files in the bucket), I change the access key. Nothing worked.

Any idea what is the problem?

Best Answer

Hi Yona,

Based on the error message that you have shared, it seems that the location 's3://{mybucket}/' seems to be the cause of the issue. Please replace {mybucket} with the actual bucket name without any flower brackets.

Even in the case of the above commands:

setenv('AWS_ACCESS_KEY_ID', <myKeyID>); 
setenv('AWS_SECRET_ACCESS_KEY', <myaccessKey>);

Please remove the angular brackets and then fill in the keyID and the access key as string type.

If all else fails, you can also use the following API for the data store:

fs = matlab.io.datastore.DsFileSet("s3://{mybucket}/", "IncludeSubFolders", true);

Hope this helps!

Thanks,

Arvind

Related Solutions

MATLAB: How can i access .mat files, stored in an encrypted amazon S3 bucket via Matlab (API Key available)

Hi David, it might be easier to use FileDatastore with the load function to do this:

% Set up S3 credentials.
setenv('AWS_ACCESS_KEY_ID', MY_ACCESS_KEY_ID); 
setenv('AWS_SECRET_ACCESS_KEY', MY_SECRET_ACCESS_KEY); 
setenv('AWS_REGION', MY_REGION);
% Construct a FileDatastore with a S3 URI.
fds = fileDatastore("s3://rdmello/*.mat", "ReadFcn", @load);
% Read from a single MAT file.
data = read(fds);
% Read all the data from all the MAT files on S3.
all_data = readall(fds);

Note here that MY_ACCESS_KEY_ID, MY_SECRET_ACCESS_KEY, and MY_REGION are strings that you need to provide in your code. See the Working with Remote Data doc page for more information about this.

Using a datastore here provides some benefits...each file on Amazon S3 is only downloaded only when needed, and you can build a tall array over it for easier scalability on parallel clusters.

MATLAB: How to load or display content of the file from AWS S3

Finally. Well following worked :

fileName = strrep(fileName,"s3://","https://s3.amazonaws.com/");
%fileName ='https://s3.amazonaws.com/{bucket}/summary.txt';
options = weboptions('ContentType','text');
data = webread(fileName, options);

Best Answer

Related Solutions

MATLAB: How can i access .mat files, stored in an encrypted amazon S3 bucket via Matlab (API Key available)

MATLAB: How to load or display content of the file from AWS S3

Related Question