Google Earth Engine – Export Monthly PRISM Precipitation Data for US Counties

exportgoogle-earth-enginegoogle-earth-engine-javascript-apiprecipitation

I'm trying to extract a monthly precipitation value from the PRISM climate dataset for each month of 2018 through 2021 (48 months total) for each US county in my shapefile (952 counties total) in Google Earth Engine. My goal is to export this as a table to then use for analyses in R. I believe I've completed the first step in the code below, but I'm unable to export the data to my Google Drive. Every time I try to export the data, the export runs for >4 hours without completion. I've tried removing the geometry portion and I've tried running this on only 2-4 months' worth of data, but my export never finishes.

Is this due to the size of the data/computer and network strength, or is there something in my code that is causing this export to continue for hours without successfully exporting?

I'm new to GEE and coding in general.


//Shapefile: 15 states, all counties
var countyList = ee.FeatureCollection(table);

print(countyList.size(), 'counties');

//Map.addLayer(countyList,{}, 'counties');
//Map.setCenter(-84.106, 42.424, 4.5);

//PRISM climate dataset
var precip = ee.ImageCollection('OREGONSTATE/PRISM/AN81m')
    .select('ppt')
    .filterBounds (countyList)
    .filterDate('2018-01', '2022-01');
    
// Specify the date range required
var startDate = ee.Date('2018-01');
var endDate = ee.Date('2022-01');
print('Observations begin on: ', startDate.format('YYYY-MM'),
  'and end on: ', endDate.format('YYYY-MM'));

// Calculate the number of months to process
var nMonths = ee.Number(endDate.difference(startDate, 'month')).round();
print('Number of months to process: ', nMonths);

//Calculate monthly precipitation for each county
var monthprecipcounty = ee.FeatureCollection(ee.List.sequence (0, nMonths.subtract(1))
  .map(function(monthoffset) {
    var month = startDate.advance(ee.Number(monthoffset), 'months')
    var monthlymean = precip
      .filterDate(month, month.advance (1, 'months'))
      .mean()
    return countyList.map(function (county) {
      var mean = monthlymean.reduceRegion({
        reducer: ee.Reducer.mean(),
        geometry: county.geometry(),
        scale: 4638.3
      })
      return county
        .set(mean)
        .set('month', month.format('yyyy-MM'))
    })
  })
  ).flatten();

//Remove geometry from data for export: I tried this but it didn't make export any simpler
//var featureCollection = monthprecipcounty.map(function(feat){
//  var nullfeat = ee.Feature(null)
//  return nullfeat.copyProperties(feat)
//})

//Export county-level precip data to Google Drive
Export.table.toDrive({
    collection: monthprecipcounty, 
    description: 'precip', 
    folder: 'this',
    fileNamePrefix: 'precip', 
    fileFormat: 'CSV',
});

https://code.earthengine.google.com/c2ae03b6ec2d7bfe5151aafac3a85a54

Best Answer

The problem is related with the geometry of complete shapefile. If you try to print monthprecipcounty Feature Collection you will get the following error message:

FeatureCollection (Error) Collection.geometry: Geometry has too many edges (2533749 > 2000000).

For avoided this error, I only considered the first 100 elements of 952 total counties and simplifying each feature geometry as follows (it was also returned features with null geometry in monthprecipcounty to eliminate the possibility of writing in the CSV file a lot of unnecessary characters to reduce processing time at the server):

//Shapefile: 15 states, all counties
var countyList = ee.FeatureCollection(table).limit(100);

print(countyList.size(), 'counties');

countyList = ee.FeatureCollection(countyList.map(function (feat) {
  
  return ee.Feature(feat.geometry().simplify({'maxError': 1}));
  //return ee.Feature(feat.geometry());
  
}));

Complete code can be obtained from here.

After running above code for 20 minutes, I got a CSV file with 4801 records (a little bit more than the 10 % of your counties). It looks as follows opened with LibreOffice Calc.

So, you can try it for complete shapefile or split it in smaller parts. You decide then.

Related Solutions

[GIS] Exporting Time Series Data using Google Earth Engine and JavaScript

You will have to make a long addition to your script to extract not from the mosaic, but from a collection for each of your indices.

Then you reduce the images with .reduceregion() and add it as a property to a feature.

After that, you create a collection of the feature so that it can be exported to a csv.

//////////////////
// Export index results as a timeseries
/////////////////

/////////////////////////////////////////////////
// Create Clip collection function
//////////////////////////////////////////////
var clipToShp = function(image){
  var clipped = image.clip(shp)
  return clipped
}

// clip all images to this shp
var clippedCollection = collection.map(clipToShp)

// Create NDVI of collection function
var ndviOfCollection = function(image){
  var ndviOfImage = image.normalizedDifference(['B5', 'B4']);
  return ndviOfImage.copyProperties(image)

}

// create a collection of the NDVI
var ndviCollection = clippedCollection.map(ndviOfCollection)

// Create NBR of collection function
var nbrOfCollection = function(image){
  var nbrOfImage = image.normalizedDifference(['B5', 'B7']);

return nbrOfImage.copyProperties(image)

}

// create a collection of the NBR
var nbrCollection = clippedCollection.map(nbrOfCollection)

// Create EVI of collection function
var eviOfCollection = function(image){
  var eviOfImage = image.expression(
      '2.5 * (nir - red) / (nir + 6 * red - 7.5 * blue + 1)',
      {
          red: image.select('B4'),    // 620-670nm, RED
          nir: image.select('B5'),    // 841-876nm, NIR
          blue: image.select('B2')    // 459-479nm, BLUE
      });


return eviOfImage.copyProperties(image)

}

// create a collection of the EVI
var eviCollection = clippedCollection.map(eviOfCollection)



// Create SAVI of collection function
var saviOfCollection = function(image){
  var saviOfImage = image.expression(
      '2.0 * (nir - red) / (nir + red + 1.0)',
      {
          red: image.select('B4'),    // 620-670nm, RED
          nir: image.select('B5'),    // 841-876nm, NIR
      });




return saviOfImage.copyProperties(image)

}

// create a collection of the SAVI
var saviCollection = clippedCollection.map(saviOfCollection)

// create a feature so we can iterate through it later
var fet = ee.Feature(shp)



// add the NDVItimeseries to the SHP
var addPropNDVI = function(img, feature){

  var newf = ee.Feature(feature)
  var featureNDVI = ee.Number(img.reduceRegion(ee.Reducer.mean(),shp))
  var theDate = img.get("DATE_ACQUIRED")//.format("Y-M-D");
  var ndviDate = ee.String("NDVI_").cat(theDate)

  return ee.Feature(newf.set(ndviDate,featureNDVI));
}


var featNDVI = ee.Feature(ndviCollection.iterate(addPropNDVI, fet));




// add the NBRtimeseries to the SHP
var addPropNBR = function(img, feature){

  var newf = ee.Feature(feature)
  var featureNBR = ee.Number(img.reduceRegion(ee.Reducer.mean(),shp))
  var theDate = img.get("DATE_ACQUIRED")
  var nbrDate = ee.String("NBR_").cat(theDate)

  return ee.Feature(newf.set(nbrDate,featureNBR));
}



var featNBR = ee.Feature(nbrCollection.iterate(addPropNBR, fet));


// add the EVI timeseries to the SHP
var addPropEVI = function(img, feature){

  var newf = ee.Feature(feature)
  var featureEVI = ee.Number(img.reduceRegion(ee.Reducer.mean(),shp))
  var theDate = img.get("DATE_ACQUIRED")
  var eviDate = ee.String("EVI_").cat(theDate)

  return ee.Feature(newf.set(eviDate,featureEVI));
}


var featEVI = ee.Feature(eviCollection.iterate(addPropEVI, fet));

// add the SAVI timeseries to the SHP
var addPropSAVI = function(img, feature){

  var newf = ee.Feature(feature)
  var featureSAVI = ee.Number(img.reduceRegion(ee.Reducer.mean(),shp))
  var theDate = img.get("DATE_ACQUIRED")
  var saviDate = ee.String("SAVI_").cat(theDate)

  return ee.Feature(newf.set(saviDate,featureSAVI));
}


var featSAVI = ee.Feature(saviCollection.iterate(addPropSAVI, fet));


//Make a collection of the features
var featureCollectionNDVI = ee.FeatureCollection([
  featNDVI])//,
var featureCollectionNBR = ee.FeatureCollection([
    featNBR])
var featureCollectionEVI = ee.FeatureCollection([
    featEVI])
var featureCollectionSAVI = ee.FeatureCollection([    
  featSAVI])

// Export Indices
Export.table.toDrive({
  collection: featureCollectionNDVI,
  description: "NDVI_Export",
  folder:"Time_Series",
  fileFormat:"CSV"
})
Export.table.toDrive({
  collection: featureCollectionNBR,
  description: "NBR_Export",
  folder:"Time_Series",
  fileFormat:"CSV"
})
Export.table.toDrive({
  collection: featureCollectionEVI,
  description: "EVI_Export",
  folder:"Time_Series",
  fileFormat:"CSV"
})
Export.table.toDrive({
  collection: featureCollectionSAVI,
  description: "SAVI_Export",
  folder:"Time_Series",
  fileFormat:"CSV"
})

======================

EDIT

Ah, I missed something.

In your question you mention:

How can I create a time series analysis (vegetation indices) of a particular shape?

When testing it, as I don't have access to your "table" I just drew a polygon, and used that as "shp". But I just noticed that your "table" is actually a Feature Collection.

If it is just one polygon that you have in your collection, you can change your early code to this.

var shp = ee.FeatureCollection(table).first().geometry();
Edit code above to change to a geometry

If it is actually a collection, you will have to modify the functions to be mapped to a collection, rather than a feature.

[GIS] Google Earth Engine: precipitation data, summing monthly data is not matching annual sum

As your code has several issues, I fixed them for adequately running it. On the other hand, as your HUC6_RMP bound is unknown for me, I assumed an arbitrary point in Idaho for only two years (1994, 1995). Complete code looks as follows:

var pt = ee.Geometry.Point (-114.71171874999999, 43.22995405445601);

var climateYEAR =  ee.ImageCollection(ee.List.sequence(1994, 1995).map(function (year){
  var date_start = ee.Date.fromYMD(year, 1, 1);
  var date_end = date_start.advance(1, "year");

  return(ee.ImageCollection('IDAHO_EPSCOR/GRIDMET')
        .select(['pr'])
        .filterDate(date_start, date_end)
        .filterBounds(pt)
          .sum()
          .set({year: year, 'system:time_start':date_start}));
}));

// filter the empty one out
var outputYearly = climateYEAR.filter(ee.Filter.listContains('system:band_names', 'constant').not())
  .sort('system:time_start').toBands();

var features = outputYearly.reduceRegions(pt, ee.Reducer.first(), 30);
print(features);

After running above code in Google Earth Engine Code Editor, I got following result for yearly precipitation (1994, 1995) in this point:

367.5252630710602 
663.8054463863373

For getting monthly precipitation values for 1994 and 1995 in same point, I used following code:

var fg_points = ee.Geometry.Point (-114.71171874999999, 43.22995405445601);

// Set years and month
var startYear = 1994;
var endYear = 1995;
var years = ee.List.sequence(startYear, endYear);
var months = ee.List.sequence(1,12);
// load the image collection
var Daily = ee.ImageCollection("IDAHO_EPSCOR/GRIDMET")
  .select(['pr']);

// make monthly summed mosaics
// loop over the years and months to get summed monthly images
var byMonth = ee.ImageCollection(ee.FeatureCollection(years.map(function(y){
  var yearCollection = Daily.filter(ee.Filter.calendarRange(y, y, 'year'));
  var byYear = ee.ImageCollection.fromImages(
    months.map(function(m) {
      var summedImage = yearCollection.filter(ee.Filter.calendarRange(m, m, 'month'))
        .reduce(ee.Reducer.sum()); 
      var date = ee.Date.fromYMD(y, m, 1).format("MM_dd_YYYY");
      return summedImage.set('system:time_start', ee.Date.fromYMD(y, m, 1)).rename(date);
      //.set('month', m).set('year', y); // eventually set year and month 
  }));
  return byYear;
})).flatten());

// filter the empty one out
var outputMonthly = byMonth.filter(ee.Filter.listContains('system:band_names', 'constant').not())
  .sort('system:time_start').toBands();

var features = outputMonthly.reduceRegions(fg_points, ee.Reducer.first(), 30);
print(features);

whose result was:

0_0_01_01_1994: 33.06542468070984
0_10_11_01_1994: 68.5323286652565
0_11_12_01_1994: 66.15042468905449
0_1_02_01_1994: 59.966766715049744
0_2_03_01_1994: 3.2267374992370605
0_3_04_01_1994: 42.53448921442032
0_4_05_01_1994: 40.94263654947281
0_5_06_01_1994: 8.581196308135986
0_6_07_01_1994: 2.761302024126053
0_7_08_01_1994: 2.5540072321891785
0_8_09_01_1994: 9.199179828166962
0_9_10_01_1994: 30.01076966524124
1_0_01_01_1995: 116.39152520895004
1_10_11_01_1995: 69.11469680070877
1_11_12_01_1995: 119.33081376552582
1_1_02_01_1995: 36.06007644534111
1_2_03_01_1995: 109.10670709609985
1_3_04_01_1995: 35.54720261693001
1_4_05_01_1995: 77.7174781858921
1_5_06_01_1995: 66.41252702474594
1_6_07_01_1995: 11.133998662233353
1_7_08_01_1995: 1.8143072128295898
1_8_09_01_1995: 8.56971126794815
1_9_10_01_1995: 12.606402099132538

The sums of these values, respectively for each year (1994, 1995), are 367.52526307106 mm and 663.805446386337; practically identical to yearly precipitation determined with your modified code. So, there is nothing wrong on the database side (GRIDMET) and your approach is not adequate for estimations of monthly means.

Best Answer

Related Solutions

[GIS] Exporting Time Series Data using Google Earth Engine and JavaScript

EDIT

[GIS] Google Earth Engine: precipitation data, summing monthly data is not matching annual sum

Related Question