8. Importing and exporting data (5 min)¶
We’ve mostly been working with data on the server side (or “in the cloud”), but sometimes you want to import data from your local computing space to GEE, or export data from GEE.
In this tutorial, we’ll import a file (a vector file) and export a clipped VIIRS-DNB image.
Our tasks in this exercise:
Convert a local shapefile for Sydney, Australia to ee object for server-side operations
Clip a VIIRS-DNB composite for 2018 to the City of Sydney
Export the clipped VIIRS image to a GDrive folder
8.1. Convert local shapefile to Earth Engine object¶
Sometimes it’s necessary to work with vector data not available in the GEE data catalog. This can often be the case with custom created geometries.
There’s a shapefile for the area of the City of Sydney Australia in the ./files/ folder in this repo. We’ll access that file and convert it to an ee object.
Of course, if you are using Google Colab, importing and exporting data locally is perhaps an unnecessary task...
To do that, we’ll use the Python json and geopandas module, which is a comprehensive library for geospatial analytics and data processing in Python, and create a helper function.
Note that the geopandas provides a warning about mixing conda channels…geospatial packages can get complex. Hopefully you are able to install smoothly, but there are resources on the geopandas doc page if you have issues.
import json # part of the base library that comes w/ Python, we dont need to install this separately
# reminder that if you are installing libraries in a Google Colab instance you will be prompted to restart your kernal
try:
import geemap, ee
import geopandas as gpd
except ModuleNotFoundError:
if 'google.colab' in str(get_ipython()):
print("package not found, installing w/ pip in Google Colab...")
!pip install geemap geopandas
else:
print("package not found, installing w/ conda...")
!conda install mamba -c conda-forge -y
!mamba install geemap -c conda-forge -y
!conda install geopandas -c conda-forge -y
import geemap, ee
import geopandas as gpd
try:
ee.Initialize()
except Exception as e:
ee.Authenticate()
ee.Initialize()
# location of shapefile locally
sydney_shp_path = "files/city_of_sydney_shapefile/sydney.shp"
# we read our shapefile in to a geopandas data frame using the geopandas.read_file method
# we'll make sure it's initiated in the EPSG 4326 CRS
sydney_gdf = gpd.read_file(sydney_shp_path, crs='EPSG:4326')
# define a helper function to put the geodataframe in the right format for constructing an ee object
def shp_to_ee_fmt(geodf):
data = json.loads(geodf.to_json())
return data['features'][0]['geometry']['coordinates']
# create the ee object
sydney = ee.Geometry.MultiPolygon(shp_to_ee_fmt(sydney_gdf))
8.2. Clip Sydney shapefile to VIIRS-DNB 2018 composite¶
8.2.1. Get VIIRS-DNB and create annual composite¶
We’ll use the avg_rad band.
viirs2018 = ee.ImageCollection("NOAA/VIIRS/DNB/MONTHLY_V1/VCMSLCFG").filterDate('2018-01-01','2018-12-31').select('avg_rad').median()
We’ll clip it to our Sydney shapefile. And intialize a map.
sydney_viirs = viirs2018.clip(sydney)
And initialize a map, setting the min/max values to stretch so we can see they dynamics of a bright city center.
sydMap = geemap.Map()
sydMap.add_basemap("SATELLITE")
sydMap.centerObject(sydney, zoom=13)
sydMap.addLayer(sydney_viirs, {'min':1, 'max':100}, 'VIIRS 2018', opacity=.75)
sydMap.addLayerControl()
sydMap
With the VIIRS resolution, even for such a small area, you can see differentiation for brightness around the dense areas, such as the downtown neighborhood by the harbor and the stadiums to the Southeast of that.
It might also be useful to note, that the native spatial resolution of VIIRS-DNB is approximately 500m. While that’s higher thand DMPS-OLS and small enough to be useful for this Area of Interest, you can see the pixel size would limit meaningful analysis or visualizing for an AOI a bit smaller than this.
8.3. Export clipped image to Gdrive¶
You may want to save a clipped file for use in other analyses. With GEE, your only immediate option is to export your data to a location in your Google Drive account (and from there you can download locally if you’d like).
You can export to GDrive useing the ee.batch.Export.image.toDrive function to create and start a task.
As with running the reduceRegion function we saw in other exercises you’ll also pass a scale parameter and set maxPixels to above the default.
Other parameters include:
explicitly set the output format to GeoTIFF.
set folder to “tmp” a folder we’ve created. If you dont set this, the file will be saved to your root.
set the description to the name of the outgoing file
task = ee.batch.Export.image.toDrive(image=sydney_viirs,
scale=30,
fileFormat='GeoTIFF',
description='Sydney_VIIRS_2018',
folder='tmp',
maxPixels=1e9)
task.start()
It may take a few moments, but our Sydney-shaped VIIRS image should now be saved in our Google Account at tmp/Sydney_VIIRS_2018.tif
Note that if you save another file to this location, your previous file will be over-written.
8.4. A note on dataset and map sharing in GEE¶
We’ve centered this tutorial on the Python API since the documentation and use examples are more difficult to find and this opens you up to working in environments outside GEE. For use of the GEE JavaScript editor, there is fairly comprehensive documentation that you can follow.
For example if you’re working with the editor you can import, export and share data using the Asset Management system, with documentation here on importing and exporting data.