Date: 10/2/2019
Updated: 11/16/20
This is first part in a series on accessing the International Debt Statistics (IDS) database through the World Bank Data API. In this part, we will query the World Bank Data API to retrieve the indicator names (including the new ones!) and location codes.
The following code in this guide will show step-by-step how to:
To start, make sure you have the following packages installed on your machine. If you aren't familiar with how to install a Python package, visit each of the linked packages for instructions.
Then, open up your preferred mode of writing Python. This could be in a Jupyter Notebook using Jupyter Lab, using a code editor (like Atom or Visual Studio) + command line, or just from the command line. Then import the required packages:
# Import packages
import requests
import json
import pandas as pd
To get a data series from the World Bank Data API, you first to use an "indicator code." For example, the indicator code for long-term external debt stock is "DT.DOD.DLXF.CD." These indicator codes can be found in the World Bank Data Catalog, but you can also use the API to explore and select indicators. In this section, we will guide you through going from a topic idea to using the World Bank Data API to select an indictor code.
More information on indicator API queries can also be found through the Data Help Desk.
To find the indicator code, we will first need to specify the API source. The World Bank Data API has numerous sources, including one specific to International Debt Statistics.
The following GET request will get us every source from the World Bank Data API. However, the request is returned in a json format that is difficult to read as is, so we will include some code to parse through it. Then we can see each source name and ID available in the World Bank Data API to find IDS.
# Get all sources from the World Bank API
sources = requests.get("http://api.worldbank.org/v2/sources?per_page=100&format=json")
sourcesJSON = sources.json()
#print(sourcesJSON) # if you want to view the JSON response as is, remove the "#" at the beginning of the print command
# Parse through the response to see the source names and ID numbers.
for i in sourcesJSON[1]:
if i["name"] == "International Debt Statistics":
print("The source ID for International Debt Statistics is " + i["id"])
else:
pass
#print(i["id"],i["name"]) # to see all the source names and IDs, remove the # at the beginning of this line
From the above result, you can see that "International Debt Statistics" has a source ID of 6. We will add that source ID to the end of our API queries.
Now that we have source ID for International Debt Statistics we can make another request to the World Bank API to receive all of the indicator names and codes associated with that source.
# Requesting the indicators for the topic External Debt
indicators = requests.get("http://api.worldbank.org/v2/indicator?format=json&source=6")
indicatorsJSON = indicators.json()
Now that we have the indicators, we can view the names and codes. The default number of results on each page is 50, but what if we want to see all the indicators on one page? Let's first print the "total" number of indicators. Then using that number we can adjust our API query.
# Print the total number of indicators
print("There are " + str(indicatorsJSON[0]["total"]) + " IDS indicators")
The above code shows the number of indicators in the External Debt topic. When submitting our new query to the API, we will take this into account by setting the "per_page" parameter to 500. This will allow us to view all the results with one query, instead of having to request multiple pages. Then we can parse through the result to explore the different indicator names and corresponding IDs.
# Get all External Debt indicators, with a per_page parameter of 500.
indicators = requests.get("http://api.worldbank.org/v2/indicator?format=json&source=6&per_page=500")
indicatorsJSON = indicators.json()
#print(indicatorsJSON) # to view ALL of the indicators as is, remove the "#" at the beginning of the print command
# Parse through the response to see the Indicator IDs and Names
for i in indicatorsJSON[1]:
IDSindicators = (i["id"],i["name"])
#print(IDSindicators) # to view the indicator ids and names, remove the "#" at the beginning of the print command
The result gives us all of the External Debt indicators and their codes. You can also view the IDS indicators and codes in their hierarchical order on our data tables. The Analytical view shows the select indicators from the IDS publication and the Standard view shows all available indicators. The indicator and code we want is "DT.DOD.DLXF.CD External debt stocks, long-term (DOD, current US$)." Before using this data, we need to understand its full definition. You can use an API query to get that information as well.
# Use the indicator code to define the "indicator" variable
indicator = "DT.DOD.DLXF.CD"
# Parse through the response to get the "sourceNote" or definition for the desired indicator
for dict_entity in indicatorsJSON[1]:
if dict_entity["id"] == indicator:
print(dict_entity["sourceNote"])
else:
pass
By using the API queries above, we were able to sort through the World Bank Data API sources to find IDS, explore the corresponding indicators, and then select one indicator of interest and find its ID and definition.
Now that we have the indicator code, we need to select debtor and creditor locations. To select a location by country, region, or income level category you will need to know its 3 letter code. This section will show you how to use similar API queries to pull the location names and codes.
For more information on location API queries visit the Data Help Desk.
When pulling the list of debtor locations, I will set the per_page result to 300.
# Requesting the locations
dlocations = requests.get("http://api.worldbank.org/v2/sources/6/country?per_page=300&format=JSON")
dlocationsJSON = dlocations.json()
# Parse through the response to see the location IDs and names
dlocations = dlocationsJSON["source"][0]["concept"][0]["variable"]
listLen = int(len(dlocations))
# Create dataframe with location values
df = pd.DataFrame(columns=["id", "value"])
for i in range(0,listLen):
code = dlocations[i]["id"]
name = dlocations[i]["value"]
df = df.append({"id":code, "value":name}, ignore_index = True)
dlocationsList = df
# See first few items in the dataframe
print(dlocationsList.head(n=10))
IDS 2021 added a 4th dimension to the data - creditor country. You can read more about that on the World Bank Data Blog. To get the full list of the creditor codes and names, you use a query like the one above, but in the get request you replace "country" with "counterpart-area."
# Requesting the locations
clocations = requests.get("http://api.worldbank.org/v2/sources/6/counterpart-area?per_page=300&format=JSON")
clocationsJSON = clocations.json()
# Parse through the response to see the location IDs and names
clocations = clocationsJSON["source"][0]["concept"][0]["variable"]
listLen = int(len(clocations))
# Create dataframe with location values
df = pd.DataFrame(columns=["id", "value"])
for i in range(0,listLen):
code = clocations[i]["id"]
name = clocations[i]["value"]
df = df.append({"id":code, "value":name}, ignore_index = True)
clocationsList = df
# See first few items in the dataframe
print(clocationsList.head(n=10))
The results from these two queries are also saved as location-codes csv*.