Google Earth Engine – Counting Pixels in Masked Raster Results in Error ‘Remote Request Too Large’

google-earth-engine

Following on from a previous question (Points to raster (image) in Google Earth Engine) I have created an Image of observations for a particular species, masked such that we are only considering pixels with a value of one (i.e. the species was observed in that pixel).

The extent of the image is necessarily large, because the observations were spread out, which means that the whole image contains many, many pixels but most are zeroes and should therefore be ignored in calculations.

All I want to do is count the number of pixels with a value of one. I tried the following code:

// Mask
var ebdMasked = ebdCountImg.mask(ebdCountImg.gt(0));

// Count pixels
var count = ebdMasked.reduceRegion({
  reducer: ee.Reducer.count(),
  geometry: roi,
  maxPixels: 1e13
  });
print(count);

Which produced the error:

Dictionary (Error)
Remote request too large (3355443200 > 83886080) for output: [<Object>].

I actually know in this case that the answer is 256, so in no way should this be too large for output. I assumed something was wrong with the mask meaning that all pixels were being considered, so I exported the masked Image to take a look. It seemed fine and, lo and behold, when I read the exported ebdMasked back in, and try again, it works as expected:

// Try again with exported layer
var ebdMasked2 = ee.Image("users/rasenior/ebdImg");

// Mask
var ebdMasked2 = ebdMasked2.mask(ebdMasked2.gt(0));

// Count pixels
var count2 = ebdMasked2.reduceRegion({
  reducer: ee.Reducer.count(),
  geometry: roi,
  maxPixels: 1e13
  });

print(count2);

Produces:

Object (1 property)
   first: 256

I'm sure I'm missing something very simple!

Full code is here:
https://code.earthengine.google.com/6608ad2279f249324ed730a56605a025

Best Answer

The error “Remote request too large …” is an unfortunate limitation — unfortunate in that it really shouldn't exist, and it's also not worded very well.

What it actually means is that you're somehow trying to compute an image that's too big (number of pixels times number of bytes needed to store each pixel). Most likely, this is due to your reprojection and reduceResolution somehow requiring computing many more pixels intermediately than the reduceRegion asked for per tile of the reduction.

Ideally, Earth Engine would automatically try computing the reduction in smaller tiles. But for now, you can specify the tileScale parameter in reductions — try tileScale: 8.

As of October 24, 2019, this error message has been changed, hopefully for the better. It now reads, for example from the above script:

Output of image computation is too large (3200.0 MiB > 80.0 MiB).
If this is a reduction, try specifying a larger 'tileScale' parameter.

Related Solutions

Google Earth Engine – Making Histogram with ui.Chart.image.histogram() in GEE

Below is a conceptual example, based on your analysis, for generating histograms by region reduction and charting the results using either ee.Array or ee.FeatureCollection objects. I prefer these methods over charting by image histogram because they are more flexible, and in the case of the ee.FeatureCollection option, the histogram table can be exported as an asset, which can help when in-browser processing times out, allowing you to make histogram charts from very large regions at small scales.

Code Editor script

/**
 * Copyright 2020 Google LLC.
 * SPDX-License-Identifier: Apache-2.0
 */

// Define a region of interest.
var geometry = ee.Geometry.Polygon(
  [[[-73.16191843371962, -37.8098361104156],
    [-73.16191843371962, -39.593041263040206],
    [-71.14043405871962, -39.593041263040206],
    [-71.14043405871962, -37.8098361104156]]], null, false);

// Display the region of interest.
Map.centerObject(geometry);
Map.addLayer(geometry);

// Filter image collection 1 and reduce to max.
var sentinel2a = ee.ImageCollection("COPERNICUS/S2")
  .filterDate('2016-10-01', '2017-03-31')
  .filterBounds(geometry)
  .filterMetadata("CLOUDY_PIXEL_PERCENTAGE",'Less_Than', 5)
  .max();

// Filter image collection 2 and reduce to median.
var sentinel2b = ee.ImageCollection("COPERNICUS/S2")
  .filterDate('2018-10-01', '2019-03-31')
  .filterBounds(geometry)
  .filterMetadata("CLOUDY_PIXEL_PERCENTAGE",'Less_Than', 5)
  .median();

// Calculate difference between two dates of normalized difference (DNDVI).
var normDif1 = sentinel2a.normalizedDifference(['B8', 'B4']);
var normDif2 = sentinel2b.normalizedDifference(['B8', 'B4']);
var dndvi = normDif1.subtract(normDif2);

// Calculate a histogram of DNDVI using a region reduction with the
// `autoHistogram()` reducer. Note that scale is set to 1000 so that the
// operation will not time out while interactive in the Code Editor browser.
// If you have a large region and a small scale, see the below section on
// exporting the result as an asset to avoid time out errors.
var hist = dndvi.reduceRegion({
  reducer: ee.Reducer.autoHistogram(),
  geometry: geometry,
  scale: 1000,
  bestEffort: true,
});

// #############################################################################
// ### HISTOGRAM CHART USING ARRAYS ###
// #############################################################################

// The result of the region reduction by `autoHistogram` is an array. Get the
// array and cast it as such for good measure.
var histArray = ee.Array(hist.get('nd'));

// Subset the values that represent the bottom of the bins and project to
// a single dimension. Result is a 1-D array.
var binBottom = histArray.slice(1, 0, 1).project([0]);

// Subset the values that represent the number of pixels per bin and project to
// a single dimension. Result is a 1-D array.
var nPixels = histArray.slice(1, 1, null).project([0]);

// Chart the two arrays using the `ui.Chart.array.values` function.
var histColumnFromArray = ui.Chart.array.values({
  array:nPixels,
  axis: 0,
  xLabels: binBottom})
  .setChartType('ColumnChart');
print(histColumnFromArray);


// #############################################################################
// ### HISTOGRAM CHART USING A FEATURE COLLECTION ###
// #############################################################################

// Cast the histogram table array as a list.
var histList = histArray.toList();

// Map over the list to create a list of features per bin and set respective
// bin bottom and number of pixels properties.
var featureList = histList.map(function(bin) {
  bin = ee.List(bin);
  var props = {
    binBottom: ee.Number(bin.get(0)),
    nPixels: ee.Number(bin.get(1))
  };
  return ee.Feature(geometry, props);
});

// Convert the feature list to a feature collection.
var featureCol = ee.FeatureCollection(featureList);

// #############################################################################
// OPTIONALLY EXPORT THE FEATURE COLLECTION

// If your region reduction times out in the browser, export the above
// feature collection as an asset. When it completes, start a new script
// that imports it and then chart it using the options following. Here is
// an example of exporting the region as an asset:

Export.table.toAsset({
  collection: featureCol,
  description: 'histogram_table',
  assetId: 'histogram_table'
});

// #############################################################################

// Chart histogram from the constructed feature collection as a line chart.
var histLineFromFc = ui.Chart.feature.byFeature({
  features: featureCol,
  xProperty: 'binBottom',
  yProperties: ['nPixels']})
  .setChartType('LineChart');
print(histLineFromFc);

// Chart histogram from the constructed feature collection as a column chart.
var histColumnFromFc = ui.Chart.feature.byFeature({
  features: featureCol,
  xProperty: 'binBottom',
  yProperties: ['nPixels']})
  .setChartType('ColumnChart');
print(histColumnFromFc);

[GIS] Counting number of unmasked pixels per image in collection using Google Earth Engine

To get to your end goal of filtering your clipped image collection by 10% or less cloud cover, I would write a function that:

counts the number of unmasked pixels in the image
counts the number of pixels in the unmasked image
calculates the percentage of masked pixels and sets this as a metadata property

I would then map this onto your image collection and filter it by the added metadata property.

Note that I'm assuming that your images are only masked by a cloud mask.


function get_cloud_cover_roi(image){ 

  var Peyto_pixelscount = image.select('ALBEDO').reduceRegion({
    reducer: ee.Reducer.count(),
    geometry: image.geometry(),
    scale: 30,
    maxPixels: 1e9
  }).get('ALBEDO')

  var npix = image.select('ALBEDO').unmask().reduceRegion({
    reducer: ee.Reducer.count(),
    geometry: image.geometry(),
    scale: 30,
    maxPixels: 1e9
  }).get('ALBEDO')

  var cloud_cover_roi = ee.Number(1)
      .subtract(ee.Number(Peyto_pixelscount).divide(npix))
      .multiply(100)

  return image.set('cloud_cover_roi', cloud_cover_roi)

}

var Peytoclip_filtered = Peytoclip
    .map(get_cloud_cover_roi)
    .filterMetadata('cloud_cover_roi', 'less_than', 10)

Best Answer

Related Solutions

Google Earth Engine – Making Histogram with ui.Chart.image.histogram() in GEE

[GIS] Counting number of unmasked pixels per image in collection using Google Earth Engine

Related Question