{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/worldbank/OpenNightLights/blob/master/onl/tutorials/mod3_7_import_export_data.ipynb)\n",
    "\n",
    "# Importing and exporting data (5 min)\n",
    "\n",
    "We've mostly been working with data on the server side (or \"in the cloud\"), but sometimes you want to import data from your local computing space to GEE, or export data from GEE.\n",
    "\n",
    "In this tutorial, we'll import a file (a vector file) and export a clipped VIIRS-DNB image.\n",
    "\n",
    "**Our tasks in this exercise:**\n",
    "1. Convert a local shapefile for Sydney, Australia to ee object for server-side operations\n",
    "2. Clip a VIIRS-DNB composite for 2018 to the City of Sydney\n",
    "3. Export the clipped VIIRS image to a GDrive folder\n",
    "\n",
    "## Convert local shapefile to Earth Engine object\n",
    "\n",
    "Sometimes it's necessary to work with vector data not available in the GEE data catalog. This can often be the case with custom created geometries.\n",
    "\n",
    "There's a shapefile for the area of the City of Sydney Australia in the `./files/` folder in this repo. We'll access that file and convert it to an `ee` object.\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "This notebook requires access to this file \"sydney.shp\". If you have cloned and downloaded this repo, then you will have this file in the \"/files/\" sub-folder of this repo and should not have to do anything more. However, if you did not download this repo, but only downloaded this notebook. OR if you are using a Google Colab instance, then you will need to make sure that you have saved \"sydney.shp\" and you are pointing the variable \"sydney_shp_path\" to the location where you saved the file.<br><br>Of course, if you are using Google Colab, importing and exporting data locally is perhaps an unnecessary task...</div>\n",
    "\n",
    "\n",
    "To do that, we'll use the Python `json` and <a href=\"https://geopandas.org/\">geopandas module</a>, which is a comprehensive library for geospatial analytics and data processing in Python, and create a helper function.\n",
    "\n",
    "Note that the geopandas provides a warning about mixing conda channels...geospatial packages can get complex. Hopefully you are able to install smoothly, but there are resources <a href=\"https://geopandas.org/install.html\">on the geopandas doc page</a> if you have issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json # part of the base library that comes w/ Python, we dont need to install this separately\n",
    "\n",
    "# reminder that if you are installing libraries in a Google Colab instance you will be prompted to restart your kernal\n",
    "\n",
    "try:\n",
    "    import geemap, ee\n",
    "    import geopandas as gpd\n",
    "except ModuleNotFoundError:\n",
    "    if 'google.colab' in str(get_ipython()):\n",
    "        print(\"package not found, installing w/ pip in Google Colab...\")\n",
    "        !pip install geemap geopandas\n",
    "    else:\n",
    "        print(\"package not found, installing w/ conda...\")\n",
    "        !conda install mamba -c conda-forge -y\n",
    "        !mamba install geemap -c conda-forge -y\n",
    "        !conda install geopandas -c conda-forge -y\n",
    "    import geemap, ee\n",
    "    import geopandas as gpd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "        ee.Initialize()\n",
    "except Exception as e:\n",
    "        ee.Authenticate()\n",
    "        ee.Initialize()\n",
    "\n",
    "# location of shapefile locally\n",
    "sydney_shp_path = \"files/city_of_sydney_shapefile/sydney.shp\"\n",
    "\n",
    "# we read our shapefile in to a geopandas data frame using the geopandas.read_file method\n",
    "# we'll make sure it's initiated in the EPSG 4326 CRS\n",
    "sydney_gdf = gpd.read_file(sydney_shp_path, crs='EPSG:4326')\n",
    "\n",
    "# define a helper function to put the geodataframe in the right format for constructing an ee object\n",
    "def shp_to_ee_fmt(geodf):\n",
    "    data = json.loads(geodf.to_json())\n",
    "    return data['features'][0]['geometry']['coordinates']\n",
    "\n",
    "# create the ee object\n",
    "sydney = ee.Geometry.MultiPolygon(shp_to_ee_fmt(sydney_gdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clip Sydney shapefile to VIIRS-DNB 2018 composite\n",
    "\n",
    "### Get VIIRS-DNB and create annual composite\n",
    "\n",
    "We'll use the `avg_rad` band."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "viirs2018 = ee.ImageCollection(\"NOAA/VIIRS/DNB/MONTHLY_V1/VCMSLCFG\").filterDate('2018-01-01','2018-12-31').select('avg_rad').median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll clip it to our Sydney shapefile. And intialize a map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "sydney_viirs = viirs2018.clip(sydney)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And initialize a map, setting the min/max values to stretch so we can see they dynamics of a bright city center."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71e1ef4c57fb45c8a40041577d6d7549",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map(center=[-33.889462622418314, 151.20275194560503], controls=(WidgetControl(options=['position', 'transparen…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sydMap = geemap.Map()\n",
    "sydMap.add_basemap(\"SATELLITE\")\n",
    "sydMap.centerObject(sydney, zoom=13)\n",
    "sydMap.addLayer(sydney_viirs, {'min':1, 'max':100}, 'VIIRS 2018', opacity=.75)\n",
    "sydMap.addLayerControl()\n",
    "sydMap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the VIIRS resolution, even for such a small area, you can see differentiation for brightness around the dense areas, such as the downtown neighborhood by the harbor and the stadiums to the Southeast of that.\n",
    "\n",
    "It might also be useful to note, that the native spatial resolution of VIIRS-DNB is approximately 500m. While that's higher thand DMPS-OLS and small enough to be useful for this Area of Interest, you can see the pixel size would limit meaningful analysis or visualizing for an AOI a bit smaller than this.\n",
    "\n",
    "## Export clipped image to Gdrive\n",
    "\n",
    "You may want to save a clipped file for use in other analyses. With GEE, your only immediate option is to export your data to a location in your Google Drive account (and from there you can download locally if you'd like).\n",
    "\n",
    "You can export to GDrive useing the `ee.batch.Export.image.toDrive` function to create and start a task.\n",
    "\n",
    "As with running the `reduceRegion` function we saw in other exercises you'll also pass a scale parameter and set maxPixels to above the default. \n",
    "\n",
    "Other parameters include:\n",
    "- explicitly set the output format to GeoTIFF.\n",
    "- set folder to \"tmp\" a folder we've created. If you dont set this, the file will be saved to your root.\n",
    "- set the description to the name of the outgoing file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = ee.batch.Export.image.toDrive(image=sydney_viirs,\n",
    "                                     scale=30,\n",
    "                                     fileFormat='GeoTIFF',\n",
    "                                     description='Sydney_VIIRS_2018',\n",
    "                                     folder='tmp',\n",
    "                                     maxPixels=1e9)\n",
    "\n",
    "task.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It may take a few moments, but our Sydney-shaped VIIRS image should now be saved in our Google Account at `tmp/Sydney_VIIRS_2018.tif`\n",
    "\n",
    "Note that if you save another file to this location, your previous file will be over-written.\n",
    "\n",
    "## A note on dataset and map sharing in GEE\n",
    "\n",
    "We've centered this tutorial on the Python API since the documentation and use examples are more difficult to find and this opens you up to working in environments outside GEE. For use of the <a href=\"https://code.earthengine.google.com/\">GEE JavaScript editor</a>, there is fairly comprehensive documentation that you can follow. \n",
    "\n",
    "For example if you're working with the editor you can import, export and share data using the Asset Management system, with <a href=\"https://developers.google.com/earth-engine/guides/asset_manager\">documentation here</a> on importing and exporting data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References:\n",
    "```{bibliography} ../references.bib\n",
    ":filter: docname in docnames\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}