[GIS] How to import images bucket from AWS S3 using python

amazon s3amazon-web-servicesnumpypython

How to import images bucket from AWS s3 using python without declaring LOCAL_PATH, i want total bucket images in python environment in array format, from one week on words i am facing this issue, please any one can help out of this.

Here is my python code

import cv2
import numpy as np
import os         
from random import shuffle 
from tqdm import tqdm      
import tensorflow as tf
import matplotlib.pyplot as plt
import tflearn
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.estimator import regression
import boto
import boto.s3.connection
import sys
import traceback    
conn =   boto.connect_s3('XXXXXXXXXXXXXXXX','XXXXXXXXXXXXXXXXXXXXXXXXX')
bucket = conn.get_bucket('dp-xray')

LOCAL_PATH='E:/'

for l in bucket_train:
    keyString = str(l.key)
    d = LOCAL_PATH + keyString
    try:
        l.get_contents_to_filename(d)
    except OSError:
        if not os.path.exists(d):
            os.mkdir(d)

def create_label(image_name):
    """ Create an one-hot encoded vector from image name """
    word_label = image_name.split('.')[-3]
    if word_label == 'Nor':
        return np.array([1,0])
elif word_label == 'Nod':
        return np.array([0,1])

def create_train_data():
training_data = []
for img in tqdm(os.listdir(TRAIN_DIR)):
    path = os.path.join(TRAIN_DIR, img)
    img_data = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
    img_data = cv2.resize(img_data, (IMG_SIZE, IMG_SIZE))
    training_data.append([np.array(img_data), create_label(img)])
shuffle(training_data)
np.save('train_data.npy', training_data)
return training_data

train_data = create_train_data()

Best Answer

Well not sure what is the exact error you are facing. Please post the log. I am sure you would have solved this by now. just to help others.

we should do something like this:

bucket = conn.Bucket(''dp-xray'')
objs = bucket.objects.all()

objs is whats you are expecting an array of object summary.

if you want image prefix then do 

response = conn.get_object(Bucket=bucket, Key=key)

key you will get in each object summary.

(not tested code but should work)

Related Question