{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/worldbank/OpenNightLights/blob/master/onl/tutorials/mod3_1_DMSP-OLS_annual_composites.ipynb)\n",
    "\n",
    "# DMSP-OLS annual composites in Google Earth Engine (5 min)\n",
    "\n",
    "This module looks at the various ways to transform and aggregate nighttime lights data.\n",
    "\n",
    "Two common processes are compositing and mosaicing.\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "In the context of remote sensing, the process of <b>making a composite</b> generally refers to taking a collection of spatially overlapping images and applying an aggregate function, also known as a \"reducer\" function, to these images to create a single composite image. For example, you may want to create an annual composite for a series of images by applying a reducer function, such as calculating the median, to the images to get a single image for the year that contains the median values per pixel.\n",
    "</div>\n",
    "\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "<b>Mosaicing</b> refers to the process of assembling different spatially located images from the same time period together to create a seamless single image. \n",
    "</div>\n",
    "\n",
    "\n",
    "In this tutorial, we'll identify and visualize annual composites in the DMSP-OLS nighttime lights series.\n",
    "\n",
    "The DMSP-OLS image collection in Google Earth Engine, <a href=\"https://eogdata.mines.edu/dmsp/downloadV4composites.html\">sourced by the Earth Observation Group</a>, Payne Institute for Public Policy, Colorado School of Mines, has been processed as a series of annual composites per year, per satellite (some years include two satellite sensors), so we dont have to compose them ourselves. See Elvidget et al. (1997) {cite}`elvidge1997mapping` for the methdology.\n",
    "\n",
    "**Our tasks in this exercise:**\n",
    "1. Load and inspect the DMSP-OLS nighttime lights Image Collection in GEE using `geemap`\n",
    "2. Search the reference table of DMSP-OLS satellites by year for particular annual composite (from 1996).\n",
    "3. Add the selected annual composite to a map.\n",
    "4. Add a 2nd annual composite from another year (2010) and create a slider panel to view and compare both.\n",
    "\n",
    "\n",
    "## Load and inspect the DMSP-OLS nighttime lights Image Collection\n",
    "\n",
    "### Initialize map object\n",
    "First, let's import `geemap` and initialize a geemap object centered on the greater Washington, DC area.\n",
    "We'll also add the default satellite (daytime) basemap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reminder that if you are installing libraries in a Google Colab instance you will be prompted to restart your kernal\n",
    "\n",
    "try:\n",
    "    import geemap, ee\n",
    "except ModuleNotFoundError:\n",
    "    if 'google.colab' in str(get_ipython()):\n",
    "        print(\"package not found, installing w/ pip in Google Colab...\")\n",
    "        !pip install geemap\n",
    "    else:\n",
    "        print(\"package not found, installing w/ conda...\")\n",
    "        !conda install mamba -c conda-forge -y\n",
    "        !mamba install geemap -c conda-forge -y\n",
    "    import geemap, ee"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ebb9f4eaa43545f1906b61178cc2103b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map(center=[38.9072, -77.0369], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(ch…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "        ee.Initialize()\n",
    "except Exception as e:\n",
    "        ee.Authenticate()\n",
    "        ee.Initialize()\n",
    "\n",
    "# set our initial map parameters for Washington, DC\n",
    "center_lat = 38.9072\n",
    "center_lon = -77.0369\n",
    "zoomlevel=10\n",
    "\n",
    "# initialize our map\n",
    "map1 = geemap.Map(center=[center_lat,center_lon], zoom=zoomlevel)\n",
    "map1.add_basemap('SATELLITE')\n",
    "\n",
    "map1.addLayerControl()\n",
    "map1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get DMSP-OLS Image Collection and review meta-data and description\n",
    "\n",
    "Recall the image collection for DMSP-OLS is located at: `NOAA/DMSP-OLS/NIGHTTIME_LIGHTS`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dmsp = ee.ImageCollection(\"NOAA/DMSP-OLS/NIGHTTIME_LIGHTS\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is the total number of images in this collection?** \n",
    "(It spans 1992-2013, but some years contain multiple images due to satellite overlap.)\n",
    "\n",
    "In the Google Earth Engine Editor, we call the `.size()` method on our collection.\n",
    "\n",
    "With `geemap` we can do the same thing, however, `.size()` will produce a size \"object\", so we have to add the extra step of using the `.getInfo()` method so it prints out to our notebook.\n",
    "\n",
    "In the GEE editor (in JavaScript):\n",
    "```\n",
    "print(dmsp.size());\n",
    "```\n",
    "\n",
    "In our Python notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "35\n"
     ]
    }
   ],
   "source": [
    "print(dmsp.size().getInfo())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TIP:** We can use Python's functional string method by adding `f` and brackets to dynamically print our collection size in a sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 35 images in this collection.\n"
     ]
    }
   ],
   "source": [
    "print(f\"There are {dmsp.size().getInfo()} images in this collection.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is the date range of our collection?**\n",
    "\n",
    "GEE has a set of methods called \"Reducers,\" which do a range of functions, such as get the sum or avg value of a collection. They are quite handy. The function `Reducer.minMax()` can be used to get a date range.\n",
    "\n",
    "\n",
    "In the GEE editor (in JavaScript):\n",
    "```\n",
    "var imgrange = dmsp.reduceColumns(ee.Reducer.minMax(), [\"system:time_start\"]);\n",
    "var start = ee.Date(imgrange.get('min'));\n",
    "var end = ee.Date(imgrange.get('max'));\n",
    "print('Date range: ', start, end);\n",
    "```\n",
    "\n",
    "Let's try this in Python..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Date range: ({'type': 'Date', 'value': 694224000000}, {'type': 'Date', 'value': 1356998400000})\n"
     ]
    }
   ],
   "source": [
    "imgrange = dmsp.reduceColumns(ee.Reducer.minMax(), [\"system:time_start\"])\n",
    "start = ee.Date(imgrange.get('min')).getInfo()\n",
    "end = ee.Date(imgrange.get('max')).getInfo()\n",
    "print(f\"Date range: {start, end}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Huh? One thing the GEE editor does is unpacks the JSON output (like a Python dictionary). \n",
    "\n",
    "We can do that here by accessing the \"value\" key from our `.getInfo()` object. Let's try it again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Date range: (694224000000, 1356998400000)\n"
     ]
    }
   ],
   "source": [
    "imgrange = dmsp.reduceColumns(ee.Reducer.minMax(), [\"system:time_start\"])\n",
    "start = ee.Date(imgrange.get('min')).getInfo()['value']\n",
    "end = ee.Date(imgrange.get('max')).getInfo()['value']\n",
    "print(f\"Date range: {start, end}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK...these numbers still don't make any sense. What's happening?\n",
    "\n",
    "### Date conversions...\n",
    "\n",
    "Tracking time in a world with many timezones is incredibly complicated. And you can imagine the nightmare scenario if applications (like airplane navigation systems) relied on arbitrary or inconsistent methods of tracking time. Therefore, software applications (including those that preprocessed our DMSP-OLS files as well as GEE) use what's called `Unix time`, which is what we see here.\n",
    "\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "    We'll spare you the <a href=\"https://en.wikipedia.org/wiki/Unix_time\">hairy details</a>, but basically, <b>Unix time a.k.a. Epoch time</b> refers to the time that has elapsed (not counting leap seconds) since the begining of the \"Unix epoch,\" which started at 00 hrs 00 minutes 00 seconds on 1 January, 1970 in Coordinated Universal Time (UTC). There are two important conventions you should assume unless stated otherwise: 1) this number is measured in milliseconds (a positive integer after Jan 1, 1970, negative before) and 2) it is in UTC time (equivalent to Greenwich Mean Time).\n",
    "</div>\n",
    "\n",
    "It's an extra step to convert this to a date that we humans understand in Python, but understanding timestamps is critical when working with temporal data -- especially data produced by satelliltes orbiting the planet, so hopefully this helps you appreciate working with data that spans time and space!\n",
    "\n",
    "To convert these Unix time values to readable dates, we'll:\n",
    "1. Divide by 1000 to convert to seconds and\n",
    "2. Use Python's handy `datetime` library to convert to datetime object using the `utcfromtimestamp` method (recall the time is in UTC!)\n",
    "3. Convert this datetime object to a string that we can read with the pattern: Year (%Y) - Month (%m) - Day (%d) Hour (%H): Minute (%M): Second (%S)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Date range: ('1992-01-01 00:00:00', '2013-01-01 00:00:00')\n"
     ]
    }
   ],
   "source": [
    "imgrange = dmsp.reduceColumns(ee.Reducer.minMax(), [\"system:time_start\"])\n",
    "start = ee.Date(imgrange.get('min')).getInfo()['value']\n",
    "end = ee.Date(imgrange.get('max')).getInfo()['value']\n",
    "\n",
    "# convert date\n",
    "from datetime import datetime\n",
    "start = datetime.utcfromtimestamp(start/1000).strftime('%Y-%m-%d %H:%M:%S')\n",
    "end = datetime.utcfromtimestamp(end/1000).strftime('%Y-%m-%d %H:%M:%S')\n",
    "print(f\"Date range: {start, end}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK! That makes sense.\n",
    "\n",
    "So to recap...\n",
    "\n",
    "In the GEE editor (in JavaScript):\n",
    "```\n",
    "var imgrange = dmsp.reduceColumns(ee.Reducer.minMax(), [\"system:time_start\"]);\n",
    "var start = ee.Date(imgrange.get('min'));\n",
    "var end = ee.Date(imgrange.get('max'));\n",
    "print('Date range: ', start, end);\n",
    "```\n",
    "\n",
    "In Python:\n",
    "```\n",
    "from datetime import datetime\n",
    "\n",
    "imgrange = dmsp.reduceColumns(ee.Reducer.minMax(), [\"system:time_start\"])\n",
    "start = ee.Date(imgrange.get('min')).getInfo()['value']\n",
    "end = ee.Date(imgrange.get('max')).getInfo()['value']\n",
    "\n",
    "start = datetime.utcfromtimestamp(start/1000).strftime('%Y-%m-%d %H:%M:%S')\n",
    "end = datetime.utcfromtimestamp(end/1000).strftime('%Y-%m-%d %H:%M:%S')\n",
    "print(f\"Date range: {start, end}\")\n",
    "```\n",
    "\n",
    "If this is a method you'll want to run more than once, you should create a function from these lines of code that only needs the image collection as a parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_date_range(img_collection):\n",
    "    imgrange = img_collection.reduceColumns(ee.Reducer.minMax(), [\"system:time_start\"])\n",
    "    start = ee.Date(imgrange.get('min')).getInfo()['value']\n",
    "    end = ee.Date(imgrange.get('max')).getInfo()['value']\n",
    "\n",
    "    start = datetime.utcfromtimestamp(start/1000).strftime('%Y-%m-%d %H:%M:%S')\n",
    "    end = datetime.utcfromtimestamp(end/1000).strftime('%Y-%m-%d %H:%M:%S')\n",
    "    print(f\"Date range: {start, end}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get DMSP-OLS annual composite for 1996\n",
    "\n",
    "**Recall from** {doc}`mod1_2_introduction_to_nighttime_light_data`:<br><a href=\"https://ngdc.noaa.gov/eog/dmsp/downloadV4composites.html\" class=\"alert-link\">NOAA`s \n",
    "National Center for Environmental Information</a> has the reference table of DMSP satellites: \n",
    "\n",
    "\n",
    "**DMSP-OLS satellites by year**\n",
    "\n",
    "|      | F10     | F12     | F14     | F15     | F16     | F18     |\n",
    "|------|---------|---------|---------|---------|---------|---------|\n",
    "| 1992 | F101992 |         |         |         |         |         |\n",
    "| 1993 | F101993 |         |         |         |         |         |\n",
    "| 1994 | F101994 | F121994 |         |         |         |         |\n",
    "| 1995 |         | F121995 |         |         |         |         |\n",
    "| 1996 |         | F121996 |         |         |         |         |\n",
    "| 1997 |         | F121997 | F141997 |         |         |         |\n",
    "| 1998 |         | F121998 | F141998 |         |         |         |\n",
    "| 1999 |         | F121999 | F141999 |         |         |         |\n",
    "| 2000 |         |         | F142000 | F152000 |         |         |\n",
    "| 2001 |         |         | F142001 | F152001 |         |         |\n",
    "| 2002 |         |         | F142002 | F152002 |         |         |\n",
    "| 2003 |         |         | F142003 | F152003 |         |         |\n",
    "| 2004 |         |         |         | F152004 | F162004 |         |\n",
    "| 2005 |         |         |         | F152005 | F162005 |         |\n",
    "| 2006 |         |         |         | F152006 | F162006 |         |\n",
    "| 2007 |         |         |         | F152007 | F162007 |         |\n",
    "| 2008 |         |         |         |         | F162008 |         |\n",
    "| 2009 |         |         |         |         | F162009 |         |\n",
    "| 2010 |         |         |         |         |         | F182010 |\n",
    "| 2011 |         |         |         |         |         | F182011 |\n",
    "| 2012 |         |         |         |         |         | F182012 |\n",
    "| 2013 |         |         |         |         |         | F182013 |\n",
    "\n",
    "In 1996, we have but one option to choose: `F121996`\n",
    "\n",
    "## Add this composite to our map\n",
    "\n",
    "Lets add this image as a layer to our map.\n",
    "\n",
    "Let's also apply the mask to areas with no data and adjust the opacity to 75%.\n",
    "\n",
    "Recall how to do this from {doc}`mod2_6_practical_exercise-image_visualization`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05d7a2a672744ebda5831f33521a8f90",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map(center=[38.9072, -77.0369], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(ch…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dmsp1996 = ee.Image(\"NOAA/DMSP-OLS/NIGHTTIME_LIGHTS/F101992\")\n",
    "\n",
    "# initialize our map\n",
    "map2 = geemap.Map(center=[center_lat,center_lon], zoom=zoomlevel)\n",
    "map2.add_basemap('SATELLITE')\n",
    "\n",
    "map2.addLayer(dmsp1996.mask(dmsp1996), {}, \"DMSP-OLS 1996\", opacity=0.75)\n",
    "\n",
    "map2.addLayerControl()\n",
    "map2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add a 2nd annual composite from another year (2010) and create a slider panel to view and compare both.\n",
    "\n",
    "#### Warning, this is based on `ipyleaflet` a Python library that does not play well with Google Colab, so this code will not work in the Google Colab environment but should on your local machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the satellite name from the reference table\n",
    "dmsp2010 = ee.Image(\"NOAA/DMSP-OLS/NIGHTTIME_LIGHTS/F182010\")\n",
    "\n",
    "map2.addLayer(dmsp2010.mask(dmsp2010), {}, \"DMSP-OLS 2010\", opacity=0.75)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall how to create a split panel slider from {doc}`mod2_6_practical_exercise-image_visualization`:\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "Warning, this is based on `ipyleaflet` a Python library that does not play well with Google Colab, so the split panel display will not work in the Google Colab environment but should on your local machine.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b4b9606be4141beb51e37b7a42654a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map(center=[38.9072, -77.0369], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(ch…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# initialize our map\n",
    "map3 = geemap.Map(center=[center_lat,center_lon], zoom=zoomlevel)\n",
    "map3.add_basemap('SATELLITE')\n",
    "\n",
    "# generate tile layers\n",
    "dmsp1996_tile = geemap.ee_tile_layer(dmsp1996.mask(dmsp1996), {}, 'DMSP-OLS 1996', opacity=0.75)\n",
    "dmsp2010_tile = geemap.ee_tile_layer(dmsp2010.mask(dmsp2010), {}, 'DMSP-OLS 2010', opacity=0.75)\n",
    "\n",
    "# create split map\n",
    "map3.split_map(left_layer=dmsp1996_tile, right_layer=dmsp2010_tile)\n",
    "map3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Great!** Now you've found and visualized an annual composite as well as learned a few helpful ways to access meta-data. This slider is cool as a visualization, but as noted in the first tutorial, the DMSP-OLS satellites do not have on-board calibration. As a result, the change from satellite to satellite (as is the case when comparing 1996 to 2010) might include sensor variations that do not represent actual observed changes in light.\n",
    "\n",
    "This is why you should be very careful when comparing or analyzing change in DMSP-OLS series directly.\n",
    "\n",
    "The way to address this is by adjusting our DMSP-OLS annual composites through a process we'll call \"intercalibration\" and this is the subject of a later tutorial.\n",
    "\n",
    "## References:\n",
    "```{bibliography} ../references.bib\n",
    ":filter: docname in docnames\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}