# Automated Data Pipelines

Automated pipelines keep Hub content current without manual intervention. They fetch, process, and publish data on a schedule, reducing maintenance burden and ensuring users always have access to fresh data.

## Why Automate?

| Benefit | Description |
|---------|-------------|
| **Consistency** | Data updates happen reliably on schedule |
| **Accuracy** | Eliminates manual upload errors |
| **Efficiency** | Frees staff from repetitive tasks |
| **Timeliness** | Fresh data available to users faster |

## ArcGIS API for Python

The `arcgis` Python package provides programmatic access to ArcGIS Online, enabling automated content management.

### Key Capabilities

| Capability | Description |
|------------|-------------|
| **Connect** | Authenticate to your AGOL organization |
| **Create** | Publish new feature layers from data files |
| **Update** | Overwrite existing hosted content |
| **Manage** | Control sharing, metadata, and permissions |

### Installation

```bash
pip install arcgis
```

### Basic Connection

```python
from arcgis.gis import GIS

# Connect to WB AGOL
gis = GIS("https://geowb.maps.arcgis.com", username, password)
```

```{warning}
Never hardcode credentials. Use environment variables or secure credential storage.
```

### Overwrite a Hosted Layer

```python
from arcgis.features import FeatureLayerCollection

# Get existing item
item = gis.content.get("item_id")
flc = FeatureLayerCollection.fromitem(item)

# Overwrite with new data
flc.manager.overwrite("updated_data.geojson")
```

## Example: OvertureLink Data Pipeline

The World Bank's **OvertureLink Data Pipeline** demonstrates how automated pipelines can build foundational data for Hubs.

| Aspect | Description |
|--------|-------------|
| **Purpose** | Process Overture Maps data into Hub-ready layers |
| **Output** | Buildings, roads, places, and other MDP layers |
| **Benefit** | Consistent, up-to-date foundational data across countries |

**Repository:** [github.com/worldbank/OvertureLink-Data-Pipeline](https://github.com/worldbank/OvertureLink-Data-Pipeline)

This pipeline extracts data from Overture Maps, processes it for specific country boundaries, and can publish directly to AGOL — providing an automated way to populate the {doc}`minimum-data-package`.

## Getting Started

### 1. Set Up Python Environment

```bash
python -m venv .venv
.venv\Scripts\activate  # Windows
pip install arcgis pandas geopandas
```

### 2. Secure Your Credentials

Use environment variables:

```python
import os
username = os.environ.get("AGOL_USERNAME")
password = os.environ.get("AGOL_PASSWORD")
```

### 3. Schedule Your Pipeline

| Platform | Method |
|----------|--------|
| **Windows** | Task Scheduler |
| **Linux/Mac** | cron |
| **Cloud** | AWS Lambda, Azure Functions, GitHub Actions |

## Related Resources

- {doc}`api-connections` — External data connections
- {doc}`minimum-data-package` — Foundational data requirements
- [ArcGIS API for Python Documentation](https://developers.arcgis.com/python/)