[GIS] Export multiple Images in one task – GEE Python API

exportgoogle-earth-enginepython

Context: I process a set of image using the Google Earth Engine Python API. I want to apply the process on a set of image and save the output (in my case export to GEE Assets).

What I have done: Looping through the image set and export each image one by one. That works well.

Problem: since I would like to apply my process over 60k images, I face the number of task restriction (3000).

Traceback (most recent call last):

ee.ee_exception.EEException: Too many tasks already in the queue (3000). Please wait for some of them to complete.

What I found: the geetools modules. However, by looking at the code, it also seems to iterate over the image collection and exports each image one by one.

Question : Is it possible to export multiple images in one task ? What are the solutions ?

Here the code:

col = ee.ImageCollection("COPERNICUS/S2") \
        .filterDate("2018-01-01", "2018-12-31") \
        .filterBounds(geometry)

# Get names of all image as a python list
image_names = get_name_collection(col).getInfo()
total = len(image_names)

# Create new image collection if don't existe
folder = "users/ab43536/masks_4_methods"
if folder not in [elt["id"] for elt in ee.data.getList({"id": "users/ab43536"})]:
    ee.data.createAsset({'type': "ImageCollection"}, folder)

# Apply process to all images
for i, name in enumerate(image_names):
    print("{:4d}/{} = {:05.2f}%   Image {}".format(i, total, i / total * 100, name))
    # The long process
    mask = computeCloudMasking(name)

    # Export image one by one
    export_image(mask, folder, getGeometryImage(ee.Image(name)),
                 name.split('/')[-1], num=i, total=total)

Where export_image is:

def export_image(image, asset_id="users/ab43536/", roi=None, name=None, num=None, total=None):
    """ Export one image to GEE Asset
    Arguments
        :param image: image to export
        :param roi=None:  specify the roi, default compute from image dimension
        :param name=None: name of the image
    """
    if roi == None: roi = getGeometryImage(image)
    if name == None: name = image.id().getInfo()
    description = "Default export"
    if num != None and total != None:
        description = "Image {} on {} equal {:05.2f} pourcent".format(num, total, num / total * 100)
    # print(description)
    assetId = asset_id + name

    # Create a task : export the result as image asset
    task = ee.batch.Export.image.toAsset(image=image.clip(roi),
                                         description=description,
                                         assetId=assetId,
                                         scale=30,
                                         region=roi.coordinates().getInfo(),
                                         )
    task.start()

Best Answer

Processing that amount of images in a stack would be too big, but you can export multiple images in one task by stacking the images, then exporting the stack.

Here's my code for the Java version, if you know how to translate:

// Stack collection to output image

var tsStack = function(image, list) {
    return ee.Image(list).addBands(image.select([bands]));
};
var stack = ee.Image(newcol.iterate(tsStack,ee.Image(0.0).toDouble())).slice(1)

Export.image.toDrive({image:stack,description:name+'_stack',folder:'yourfolder', region:roi, scale:scale, crs:crs})
Related Question