Google Earth Engine – Exporting Many Processed Images

batchexportgoogle-earth-engineimagejavascript

I am new to Google Earth Engine and with the help of GIS community, I was able to learn really quick. So, kudos to the GIS community.

But I am having a bit of an issue. I am currently exporting entire MODIS Lai band image collection for US continent. My script is working, but the problem is that it's taking forever to export all the images. It's been more than a day and all my tasks are still running, I barely got 10% of my Image Collection.

Originally, I wanted to get all images (from 2002 to present) in one go but if I do that then I am getting an error from GEE. it seems GEE cannot generate that many tasks at once. The number images in modis dataset are around 1630. So, instead all the images in one go I am using filterDate to export the images in parts of 2 years.

// import the modis dataset as well as DEM or shape file
// to get the exact outline of US continent using Image.mask()
var modis = ee.ImageCollection("MODIS/006/MCD15A3H");
var dem = ee.Image("users/username/dem_us_continent");

// select the "Lai" band
var modis_lai = modis.select('Lai')
                .filterDate('2018-01-01','2020-01-01');

// Get the List of all Images in ImageCollection
var size = modis_lai.size().getInfo(); // Total no. of images (around 1630)
var modis_lai_list = modis_lai.toList(size);

// Get the geometry coordinates of dem
// Properties of dem(system:footprint)
//                      type: LinearRing
//                      coordinates: List (17 elements)
var dem_coordinates = dem.get('system:footprint');

// Iterate over all the Images to Export them in the Drive
for (var n=0; n<size; n++) {
  // Get the nth index image from ImageCollection
  var image = ee.Image(modis_lai_list.get(n));

  // Set the file name based on the Dates
  var fileName = ee.Date(image.get('system:time_start'))
              .format('YYYY-MM-DD'); // For continus date: 2002-07-185
  fileName = fileName.getInfo();

  // Masked the image with the DEM to get the US Continent Area
  var masked_image = image.mask(dem);

  // Export the image to the Drive
  Export.image.toDrive({
    image: masked_image,
    description: fileName,
    crs: 'EPSG:4326',       //Projection system of download
    scale: 1000,     //Change the scale of download to matching resolution in meters.
    folder:"GEE_Data",
    region: dem_coordinates,
    dimensions: '8930x3816',
    maxPixels: 2000000000
  });
}

// Display the last MODIS masked_image in the for loop
Map.addLayer(masked_image);

To run all the tasks at once, I have used below script in web-browser's developer mode(console).

/**
 * Batch execute GEE Export task
 *
 * Once the script ran/executed,
 * export tasks will be generated and run button will show-up in Tasks section.
 *   
 * Afterwards press F12 get into console and paste the below script in it,
 * and press enter or run the below script in console.
 * All the tasks will start automatically. 
 * (Firefox and Chrome are supported.)
 */


function runTaskList(){
    var tasklist = document.getElementsByClassName('task local type-EXPORT_IMAGE awaiting-user-config');
    for (var i = 0; i < tasklist.length; i++)
            tasklist[i].getElementsByClassName('run-button')[0].click();
}

function confirmAll() {
    var ok = document.getElementsByClassName('goog-buttonset-default goog-buttonset-action');
    for (var i = 0; i < ok.length; i++)
        ok[i].click();
}

runTaskList();
confirmAll();

I wanted to know, if I am doing anything wrong here or not. Because it's taking excruciating amount of time to just export all images in my drive. Thanks in advance.

Best Answer

You might not want to hear this but essentially this is not what Google Earth Engine is designed for. It is based around the philosophy of sending your code to the data and then downloading the results.

You can export up to a point but then GEE will put limits on your download task(s).

It required an adjustment for a lot of old timers like myself (we used to download data in bulk from NASA etc.), so I understand where you are coming from, but now I am a huge fan. There is no need to store and track all the data locally, and GEE has huge compute power available for free!

Related Question