Google Earth Engine – Downloading All Images in an ImageCollection to Local Computer from Google Earth Engine

batchdownloadexportgoogle-earth-engineimage

Based on Filter Sentinel-3 data based on cloud cover over a region of interest, I managed to get an ImageColection containing the name of hundreds of Sentinel-3 images with specific cloud cover over a region of interest. Now I wish to ask about downloading this dataset as I need to process them in my local computer.

I see that GEE provides three options for downloading/exporting images:
GoogleDrive: which I don't think has enough space for my application
Google Cloud Space: which I think is not free
GEE asset: which is vague to me how it works and if it is enough for my application or not.

How do I download all images in an ImageCollection to my local computer?

Can I do that with GE asset?

If yes, how?

if no, is there any other way to batch download the images in an ImageCollection from LPDAAC or Scihub Copernicus or these kind of source websites?

Best Answer

How do I download all images in an ImageCollection to my local computer?

Earth Engine does not currently offer a prebuilt bulk direct download option. However, you could write a Python or JavaScript program that requests a series of download URLs using ee.Image.getDownloadURL. These requests have more limitations on time and size than batch tasks (Export.*) but can transfer data direct from EE to your computer without a stop at any cloud storage service.

Can I do that with GE asset?

No, because that would be just creating another asset in Earth Engine's own storage just like the one you started with.

if no, is there any other way to batch download the images in an ImageCollection from LPDAAC or Scihub Copernicus or these kind of source websites?

Perhaps. Earth Engine's assets generally have image IDs from the original data provider, which you can use to match up images within EE to those from the provider's site, but every provider is a little different and it'll depend entirely on them whether it would be easy to download a list of IDs.

Related Solutions

Google Earth Engine Sentinel-2 – How to Filter Sentinel-2 Images Based on Cloud Cover Over a Region of Interest

Cloud score algorithm is for Landsat scenes. You must use another algorithm to identify clouds. Since QA60 contains dense clouds information, you can use this band.

I posted before how to convert QA band by bits in Cloud mask for Landsat8 on Google Earth Engine, the approach is the same, so you code will mutate to:

//the parameter for sentiel-2 images for clouds is
//CLOUDY_PIXEL_PERCENTAGE
var ic =ee.ImageCollection('COPERNICUS/S2');

// A polygon representing the roi.
// my_polygon es a random polygon over Bogota Colombia (very cloudy zone)
var geometry = my_poligon;

var c = ic.filterBounds(geometry);

var getQABits = function(image, start, end, newName) {
    // Compute the bits we need to extract.
    var pattern = 0;
    for (var i = start; i <= end; i++) {
       pattern += Math.pow(2, i);
    }
    // Return a single band image of the extracted QA bits, giving the band
    // a new name.
    return image.select([0], [newName])
                  .bitwiseAnd(pattern)
                  .rightShift(start);
};

// A function to mask out cloudy pixels.
var clouds = function(image) {
  // Select the QA band.
  var QA = image.select(['QA60']);
  // Get the internal_cloud_algorithm_flag bit.
  return getQABits(QA, 10,10, 'cloud');
  // Return an image masking out cloudy areas.
};

var withCloudiness = c.map(function(image) {
  var cloud = clouds(image);
  var cloudiness = cloud.reduceRegion({
    reducer: 'mean', 
    geometry: geometry, 
    scale: 10,
    crs:'EPSG:32719'
  });
  return image.set(cloudiness);
});

var filteredCollection = withCloudiness.filter(ee.Filter.lt('cloud', 0.1));
print(filteredCollection);                

//my image with less clouds over my zone of interest                   
var my_image = ee.Image(filteredCollection.filterBounds(centroid)
    .filterDate('2019-01-01', '2019-06-30')
    .sort('CLOUDY_PIXEL_PERCENTAGE')
    .first());

NOTE: Now the filter isn't less than 10(%) of clouds, since the cloud image is 0 (no cloud) and 1 (cloud) use the fractional notation (0.1) to filter.

Finally, is right to use 'CLOUDY_PIXEL_PERCENTAGE' instead of 'cloud' for filter the less cloudy image over your study area? If you use it, you're filtering using the whole image, not your study area. You must consider changing the last sort operation

Google Earth Engine – Handling Null Inside .map in Google Earth Engine

There is not a great way to skip an image in the map function. However, you can use the ee.Algorithms.If function to control what happens when null is encountered. In your case, if all pixels in the region are masked, the reducer will return null. Instead of trying to do math operations with null, you can return a masked image (no attempt to normalize). Additionally, set an 'isNull' property so that you can filter on it once the normalize function has been mapped over the collection. Here is ee.Algorithms.If implemented in your snippet:

var normalize = function(Im){
  var time = Im.get("system:time_start");
  var percentiles = Im.reduceRegion({
    reducer:ee.Reducer.percentile([10,90],["10","90"]),
    crs:proj.crs,
    crsTransform:proj.transform,
    maxPixels:1000000000
  });
  
  var p10 = ee.Number(percentiles.get("band_10"));
  print(p10,"p10");
  var p90 = ee.Number(percentiles.get("band_90"));
  print(p90,"p90");
  
  var outImg = ee.Image(ee.Algorithms.If(
    p10, 
    Im.subtract(p10).divide(p90.subtract(p10)).set('isNull', false),
    ee.Image(0).selfMask().set('isNull', true)
  ));
  
  return outImg.set({"system:time_start":time});
};

If you wanted to filter the null images out of the collection do this:

var normCol = imgCol
  .select(["B1"], ["band"])
  .map(normalize);
  
var normColNotNull = normCol.filter(ee.Filter.eq('isNull', false));

Note that ee.Algorithms.If should be used with caution because it will always evaluate both the true and false alternatives, which takes extra time and resources.

Best Answer

Related Solutions

Google Earth Engine Sentinel-2 – How to Filter Sentinel-2 Images Based on Cloud Cover Over a Region of Interest

Google Earth Engine – Handling Null Inside .map in Google Earth Engine

Related Question