Open Data Cube – Indexing USGS Data Collection 2

Indexing USGS Landsat Collection 2 Level 2 datasets into Open Data Cube

Following the suggestion in the above URL, I tried to index USGS data into the Open Data Cube. Product definition was successfully added, however when I tried to index using stac-to-dc, I received the following error:

command used:

docker-compose exec jupyter stac-to-dc --catalog-href=https://landsatlook.usgs.gov/stac-server/ --rewrite-assets=https://landsatlook.usgs.gov/data/,s3://usgs-landsat/ --bbox=91.81029362088168,21.371284249219112,92.09814642753065,21.537368083932826 --collections=landsat-c2l2-sr --datetime=2021-01-01/2021-01-31

exception:

Indexing from STAC API...
02/21/2022 04:03:36: ERROR: Failed to handle item LE07_L2SP_136045_20210113_20210208_02_T1_SR with exception Failed to create dataset with error No matching Product found for dataset 0e138867-8b65-5712-9216-395e7af280ad
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_136045_20210113_20210208_02_T1_SR
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 158, in stac_api_to_odc
    _ = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 106, in process_item
    index_update_dataset(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 172, in index_update_dataset
    raise IndexingException(
odc.apps.dc_tools.utils.IndexingException: Failed to create dataset with error No matching Product found for dataset 0e138867-8b65-5712-9216-395e7af280ad
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_136045_20210113_20210208_02_T1_SR
02/21/2022 04:03:36: ERROR: Failed to handle item LC08_L2SP_136045_20210121_20210307_02_T1_SR with exception Failed to create dataset with error No matching Product found for dataset 7c767047-c6fe-5ca1-84b2-82a671331700
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LC08_L2SP_136045_20210121_20210307_02_T1_SR
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 158, in stac_api_to_odc
    _ = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 106, in process_item
    index_update_dataset(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 172, in index_update_dataset
    raise IndexingException(
odc.apps.dc_tools.utils.IndexingException: Failed to create dataset with error No matching Product found for dataset 7c767047-c6fe-5ca1-84b2-82a671331700
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LC08_L2SP_136045_20210121_20210307_02_T1_SR
02/21/2022 04:03:36: ERROR: Failed to handle item LE07_L2SP_135045_20210122_20210217_02_T1_SR with exception Failed to create dataset with error No matching Product found for dataset 80c2ed31-24e1-5e39-a075-290db3d231b4
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_135045_20210122_20210217_02_T1_SR
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 158, in stac_api_to_odc
    _ = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 106, in process_item
    index_update_dataset(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 172, in index_update_dataset
    raise IndexingException(
odc.apps.dc_tools.utils.IndexingException: Failed to create dataset with error No matching Product found for dataset 80c2ed31-24e1-5e39-a075-290db3d231b4
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_135045_20210122_20210217_02_T1_SR
02/21/2022 04:03:36: ERROR: Failed to handle item LE07_L2SP_135045_20210106_20210201_02_T1_SR with exception Failed to create dataset with error No matching Product found for dataset a55a7a56-2880-52d9-ab3b-24a62aeeca30
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_135045_20210106_20210201_02_T1_SR
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 158, in stac_api_to_odc
    _ = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 106, in process_item
    index_update_dataset(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 172, in index_update_dataset
    raise IndexingException(
odc.apps.dc_tools.utils.IndexingException: Failed to create dataset with error No matching Product found for dataset a55a7a56-2880-52d9-ab3b-24a62aeeca30
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_135045_20210106_20210201_02_T1_SR
02/21/2022 04:03:36: ERROR: Failed to handle item LC08_L2SP_136045_20210105_20210309_02_T1_SR with exception Failed to create dataset with error No matching Product found for dataset 07e8baa2-50ea-5a22-bcb1-bf3a74524a87
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LC08_L2SP_136045_20210105_20210309_02_T1_SR
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 158, in stac_api_to_odc
    _ = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 106, in process_item
    index_update_dataset(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 172, in index_update_dataset
    raise IndexingException(
odc.apps.dc_tools.utils.IndexingException: Failed to create dataset with error No matching Product found for dataset 07e8baa2-50ea-5a22-bcb1-bf3a74524a87
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LC08_L2SP_136045_20210105_20210309_02_T1_SR
02/21/2022 04:03:36: ERROR: Failed to handle item LE07_L2SP_136045_20210129_20210224_02_T1_SR with exception Failed to create dataset with error No matching Product found for dataset 0eb2aca9-0445-5675-b6c6-2853fed3358c
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_136045_20210129_20210224_02_T1_SR
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 158, in stac_api_to_odc
    _ = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/stac_api_to_dc.py", line 106, in process_item
    index_update_dataset(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 172, in index_update_dataset
    raise IndexingException(
odc.apps.dc_tools.utils.IndexingException: Failed to create dataset with error No matching Product found for dataset 0eb2aca9-0445-5675-b6c6-2853fed3358c
 The URI was https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LE07_L2SP_136045_20210129_20210224_02_T1_SR
Added 0 Datasets, failed 6 Datasets

Any thoughts?

Best Answer

I was able to do this just now.

I ran this to add the products:

datacube product add https://raw.githubusercontent.com/opendatacube/datacube-dataset-config/main/products/lsX_c2l2_sr.odc-product.yaml

And this to do the indexing:

stac-to-dc --catalog-href=https://landsatlook.usgs.gov/stac-server/ --rewrite-assets=https://landsatlook.usgs.gov/data/,s3://usgs-landsat/ --bbox=91.81029362088168,21.371284249219112,92.09814642753065,21.537368083932826 --collections=landsat-c2l2-sr --datetime=2021-01-01/2021-01-31

With the output:

Indexing from STAC API... Added 6 Datasets, failed 0 Datasets

Best Answer

Related Solutions

Open Data Cube – Solving Data Indexing Issues from DEAfrica STAC

Open Data Cube USGS Datasets – Indexing USGS Surface Temperature Datasets into Open Data Cube

Related Question