{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/worldbank/OpenNightLights/blob/master/onl/tutorials/mod3_6_making_VIIRS_annual_composites.ipynb)\n",
    "\n",
    "\n",
    "# Making simple VIIRS-DNB annual composites (5 min)\n",
    "\n",
    "Temporal composites, such as annual composites, are created for satellite data for a variety of reasons. Sometimes it can be desirable to visualize or conduct trend analysis on a smoother, less dense time series, especially if you're comparing multiple data sources and need a common temporal unit of analysis, like a year.\n",
    "\n",
    "Often the aim is to reduce the noise that occurs with shorter time periods under the intuition that noise (e.g. via stochastic processes) will be minimalized, or \"canceled out\", when data are aggregated over longer time periods, whereas the true signal will be preserved or even strengthened relative to the noise levels.\n",
    "\n",
    "We've been working with DMSP-OLS data to this point. For this exercise, we'll work with the VIIRS-DNB data so you can get familiar with this source. Refer to {doc}`mod1_2_introduction_to_nighttime_light_data` for a refresher on the VIIRS-DNB dataset. \n",
    "\n",
    "We will create a simple annual composite of VIIRS-DNB by aggregating monthly composites in Google Earth Engine (GEE) using the `Reduce()` function on an ImageCollection.\n",
    "\n",
    "**Our tasks in this exercise:**\n",
    "1. Brief background on the VIIRS-DNB stray-light monthly composite data.\n",
    "2. Filter a selection of VIIRS-DNB data for the year 2015.\n",
    "3. Create an annual composite of 2015 using the Reduce function.\n",
    "4. Create a time series of annual composites from VIIRS-DNB monthly composites for 2015-2019.\n",
    "\n",
    "## Background on VIIRS-DNB monthly composites\n",
    "\n",
    "For this exercise, we're going to work with the VIIRS-Day/Night Band (DNB) image collection that has been corrected for stray-light and filtered for data quality, which includes cloud coverage. The daily images are aggregated into monthly composites. In Google Earth Engine, this Image Collection is: `NOAA/VIIRS/DNB/MONTHLY_V1/VCMSLCFG`\n",
    "\n",
    "<div class=\"alert alert-success\">You can read more about the data and the methodology for cleaning and creating composites <a href=\"https://eogdata.mines.edu/download_dnb_composites.html\">here</a></div>\n",
    "\n",
    "Read more about the process to correct for stray light in this paper {cite}`mills2013viirs`.\n",
    "\n",
    "**Note:** these monthly composites are not filtered to screen out light from auroras, fires, boats or other temporal lights; however, annual composites are available that have this correction through 2015 <a href=\"https://eogdata.mines.edu/download_dnb_composites.html\">here</a>\n",
    "\n",
    "Cleaning and creating composites using daily Sensor Data Records made available through the partnership with the World Bank, University of Michigan, and NOAA will also be the subject of a more advanced Python library as part of this Open Night Lights platform (forthcoming).\n",
    "\n",
    "## Get ImageCollection for 2015 VIIRS-DNB data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reminder that if you are installing libraries in a Google Colab instance you will be prompted to restart your kernal\n",
    "\n",
    "try:\n",
    "    import geemap, ee\n",
    "except ModuleNotFoundError:\n",
    "    if 'google.colab' in str(get_ipython()):\n",
    "        print(\"package not found, installing w/ pip in Google Colab...\")\n",
    "        !pip install geemap\n",
    "    else:\n",
    "        print(\"package not found, installing w/ conda...\")\n",
    "        !conda install mamba -c conda-forge -y\n",
    "        !mamba install geemap -c conda-forge -y\n",
    "    import geemap, ee"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "there are 12 images in this collection\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "        ee.Initialize()\n",
    "except Exception as e:\n",
    "        ee.Authenticate()\n",
    "        ee.Initialize()\n",
    "\n",
    "# get the 2015 image collection, we're using the \"avg_rad\" band\n",
    "viirs2015 = ee.ImageCollection(\"NOAA/VIIRS/DNB/MONTHLY_V1/VCMSLCFG\").filterDate(\"2015-01-01\",\"2015-12-31\").select('avg_rad')\n",
    "\n",
    "# as a gut-check, there should be 12 images in this collection!\n",
    "print(f\"there are {viirs2015.size().getInfo()} images in this collection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an annual composite using the median of these images\n",
    "\n",
    "Recall in {doc}`mod3_1_DMSP-OLS_annual_composites` we were introduced to the `Reducer` function. In GEE, ImageCollections also have a few handy reducer functions built in, including `median` and `mean` among others. As the names suggest, these functions reduce a given ImageCollection to a single Image by calculating the median or the mean.\n",
    "\n",
    "#### We'll reduce our ImageCollection of 12 images from 2015 to a single Image by calculating the median. Then we'll initiate a geemap Map object and visualize our layer as we've done before.\n",
    "\n",
    "We haven't \"been\" to South America in our tutorials yet, so we'll center our image on Sao Paulo, Brazil.\n",
    "\n",
    "We've already selected our band \"avg_rad\" when filtering by date, so we can just use our existing variable, `viirs2015`.\n",
    "\n",
    "As we did with our DMSP-OLS images, let's also apply the mask to ignore cells in our raster that contain no data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9003c96c86194eb9998a7e422c4111e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map(center=[-23.54, -46.63], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(child…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "viirs2015med = viirs2015.median()\n",
    "\n",
    "# iniatialize map on Sao Paulo\n",
    "lat = -23.54\n",
    "lon = -46.63\n",
    "\n",
    "# initialize our map\n",
    "map1 = geemap.Map(center=[lat,lon], zoom=8)\n",
    "\n",
    "map1.add_basemap('SATELLITE')\n",
    "\n",
    "map1.addLayer(viirs2015med.mask(viirs2015med), {}, \"VIIRS-DNB 2015 (monthly med)\")\n",
    "map1.addLayerControl()\n",
    "map1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a time series of annual composites from 2015 to 2019\n",
    "\n",
    "If you want to create a time series of annual composites, you can filter and reduce the collection as we did for 2015, but for each year you want data.\n",
    "\n",
    "Thinking programmatically, of course, don't do this manually. We'll use the Earth Engine `list` and `map` functions to iterate through our ImageCollection and create annual composites for each year.\n",
    "\n",
    "### First, we'll use Earth Engine's `list` method to create a list object of years:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "our list has 5 years in it\n"
     ]
    }
   ],
   "source": [
    "# define our start and end years\n",
    "start = 2015\n",
    "end = 2019\n",
    "\n",
    "years = ee.List.sequence(start, end)\n",
    "\n",
    "print(f\"our list has {years.size().getInfo()} years in it\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Then we'll create a function to filter Images to a given year and reduce them, producing an annual composite that we'll map to each year in our list and create a new image collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "colID = \"NOAA/VIIRS/DNB/MONTHLY_V1/VCMSLCFG\"\n",
    "\n",
    "def viirs_annual_median_reduce(year):\n",
    "    return ee.ImageCollection(colID).filter(\n",
    "        ee.Filter.calendarRange(year,year,\"year\")).select(\"avg_rad\").median().set('year',year)\n",
    "\n",
    "# map function to each year in our list\n",
    "yearComps = ee.ImageCollection.fromImages(years.map(viirs_annual_median_reduce))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we can filter on our image collection on any year to get that composite. We could add multiple years to our map object for comparison.\n",
    "\n",
    "Let's create a new map Object and later will add a slide window to it.\n",
    "\n",
    "In a later tutorial we'll learn how to make time series plots and compare images using histograms. For now, we'll just add each annual composite as a layer to our map using a simple Python loop. (and we'll mask each layer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "map2 = geemap.Map(center=[lat,lon], zoom=8)\n",
    "\n",
    "map2.add_basemap('SATELLITE')\n",
    "\n",
    "# add each layer\n",
    "for year in range(start,end+1):\n",
    "    img = yearComps.filterMetadata(\"year\",\"equals\",year).first() #there's only one image, but we extract from collection\n",
    "    map2.addLayer(img.mask(img), {}, f\"VIIRS-DNB {year}\", opacity=.75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9638e2a277d242ad963e9003d7e54d69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map(center=[-23.54, -46.63], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(child…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "map2.addLayerControl()\n",
    "map2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "Warning, this is based on `ipyleaflet` a Python library that does not play well with Google Colab, so the split panel display will not work in the Google Colab environment but should on your local machine.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff4844b8c5e642b296cd5bb082d91eb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map(center=[-23.54, -46.63], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(child…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create a split panel map\n",
    "left_layer = geemap.ee_tile_layer(yearComps.filterMetadata(\"year\",\"equals\",2015), {},\n",
    "                                  'VIIRS-DNB 2015', opacity=0.75)\n",
    "right_layer = geemap.ee_tile_layer(yearComps.filterMetadata(\"year\",\"equals\",2019), {},\n",
    "                                   'VIIRS-DNB 2019', opacity=0.75)\n",
    "\n",
    "map3 = geemap.Map(center=[lat,lon], zoom=8)\n",
    "map3.add_basemap('SATELLITE')\n",
    "map3.split_map(left_layer=left_layer, right_layer=right_layer)\n",
    "map3.addLayerControl()\n",
    "map3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "VIIRS-DNB has noticeably higher resolution than the DMSP-OLS, so differences in light spatial distribution are visible when investigating dynamics over even a few years.\n",
    "\n",
    "Visualizing these changes is interesting, but later in the tutorial we'll look at carrying out operations on our images that facilitate analytical study as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References:\n",
    "```{bibliography} ../references.bib\n",
    ":filter: docname in docnames\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}