Automated Data Pipelines#
Automated pipelines keep Hub content current without manual intervention. They fetch, process, and publish data on a schedule, reducing maintenance burden and ensuring users always have access to fresh data.
Why Automate?#
Benefit |
Description |
|---|---|
Consistency |
Data updates happen reliably on schedule |
Accuracy |
Eliminates manual upload errors |
Efficiency |
Frees staff from repetitive tasks |
Timeliness |
Fresh data available to users faster |
ArcGIS API for Python#
The arcgis Python package provides programmatic access to ArcGIS Online, enabling automated content management.
Key Capabilities#
Capability |
Description |
|---|---|
Connect |
Authenticate to your AGOL organization |
Create |
Publish new feature layers from data files |
Update |
Overwrite existing hosted content |
Manage |
Control sharing, metadata, and permissions |
Installation#
pip install arcgis
Basic Connection#
from arcgis.gis import GIS
# Connect to WB AGOL
gis = GIS("https://geowb.maps.arcgis.com", username, password)
Warning
Never hardcode credentials. Use environment variables or secure credential storage.
Overwrite a Hosted Layer#
from arcgis.features import FeatureLayerCollection
# Get existing item
item = gis.content.get("item_id")
flc = FeatureLayerCollection.fromitem(item)
# Overwrite with new data
flc.manager.overwrite("updated_data.geojson")
Example: OvertureLink Data Pipeline#
The World Bank’s OvertureLink Data Pipeline demonstrates how automated pipelines can build foundational data for Hubs.
Aspect |
Description |
|---|---|
Purpose |
Process Overture Maps data into Hub-ready layers |
Output |
Buildings, roads, places, and other MDP layers |
Benefit |
Consistent, up-to-date foundational data across countries |
Repository: github.com/worldbank/OvertureLink-Data-Pipeline
This pipeline extracts data from Overture Maps, processes it for specific country boundaries, and can publish directly to AGOL — providing an automated way to populate the Minimum Data Package.
Getting Started#
1. Set Up Python Environment#
python -m venv .venv
.venv\Scripts\activate # Windows
pip install arcgis pandas geopandas
2. Secure Your Credentials#
Use environment variables:
import os
username = os.environ.get("AGOL_USERNAME")
password = os.environ.get("AGOL_PASSWORD")
3. Schedule Your Pipeline#
Platform |
Method |
|---|---|
Windows |
Task Scheduler |
Linux/Mac |
cron |
Cloud |
AWS Lambda, Azure Functions, GitHub Actions |