The previous chapter sought to provide insights into the patterns of coverage gaps currently found in ESG data. This section seeks to provide further insight into why these gaps occur.
To better understand the nature of data gaps we consulted data management experts at the World Bank to develop a list of explanatory factors which are described below. These “explanations” are defined such that they can be used to consistently “tag” indicators to which they apply. They are particularly relevant to time series statistics. Using metadata in the World Bank API and interviews with members of the data management team, we examined each indicator through the lens of these explanations and recorded the results as dummy (i.e., yes/no) values in a metadata database. It is possible that multiple explanations apply to any given indicator.
The results of this tagging exercise were used to build a database along with other relevant metadata. The final database assembled for analysis has a total of 48 variables for 134 indicators.
The discusion below describes each of the explanations in detail. Each description includes a coverage heat map from the previous section which identifies the indicators “tagged” underneath it, showing how the explanation may impact gaps in data coverage.
The World Bank API defines 217 countries and economies. Indicators that had observations for 2018 or later for at least 196 countries (90%) were considered “non gap” indicators and were not tagged with any further explanations.
Figure 3.1: Ideal Case
While most major World Bank time series databases are updated on regular schedules, it is common for individual indicators and occasionally entire databases to be discontinued. This can occur if the original provider or program discontinues support for the data, or if policies, programs, or methodologies evolve such that the data are no longer relevant. In these cases, the time series is no longer updated with recent observations, and the series becomes increasingly obsolete. Deprecated indicators in the WDI are typically removed and are available only in the WDI Archives. Accordingly, it is not possible to provide a heatmap of these indicators.
Table 3.1: Dropped or Deprecated
| Indicator | Code |
|---|---|
| PM10, country level (micrograms per cubic meter) | EN.ATM.PM10.MC.M3 |
| GEF benefits index for biodiversity (0 = no biodiversity potential to 100 = maximum) | ER.BDV.TOTL.XQ |
| Overall surplus/deficit, excluding current grants (current LCU) | GB.BAL.OVRX.CN |
| Fiscal balance, cash surplus/deficit (current US$) | GC.BAL.CASH.CD |
| Open Budget Index Overall Country Score | IBP.OBI.XQ |
| Average duration of power outages (hours) | IC.ELC.OUTG.HR |
| Telephone mainlines (per 1,000 people) | IT.MLT.MAIN.P3 |
| Internet users (per 1,000 people) | IT.NET.USER.P3 |
| Genuine savings: energy depletion (% of GDP) | NY.GEN.DNGY.GD.ZS |
| Gross national savings, public (current US$) | NY.GNS.PUBL.CD |
| Tuberculosis prevalence rate, low uncertainty bound (per 1000,000 population, WHO) | SH.TBS.PREV.LW |
| Labor force with secondary education (% of total) | SL.TLF.SECO.ZS |
| Labor force with tertiary education (% of total) | SL.TLF.TERT.ZS |
| Depth of the food deficit (kilocalories per person per day) | SN.ITK.DFCT |
| Export product concentration index | TX.CONC.IND.XQ |
Definition: the indicator only exists in a discontinued or “archived” database, including:
Notes: indicators tagged with this explanation are considered inactive and remaining explanations are not considered.
In other cases, an indicator may remain in an active database but fall out of active maintenance. This may be the case as a precursor to Explanation A above, or it may be attributable to lapsed oversight, i.e., there is no longer a person or team responsible for the indicator (e.g., program termination, department reorganization, personnel changes). In these cases as well, the time series becomes increasingly obsolete as it is no longer updated. Some indicators that fit this definition however, may simply be subject to very long update cycles of 4-5 years or more, meaning that they may still be actively maintained.
Figure 3.2: No Longer Updated
Definition: the indicator has no observations for any country for the last four years (2015-2018)
Notes: indicators tagged with this explanation are considered inactive and remaining explanations are not considered.
Some indicators by their nature may be especially time consuming to produce. For instance, if an indicator relies on administrative data or other underlying raw information, it cannot be calculated until that source is published. These source dependencies may be subject to their own production schedules and timeliness issues. The effect may be that an indicator may only be available significantly later than the time period of its observations. These delays may be compounded if multiple dependencies are involved. In Figure 3.3 indicators that embody this characteristic exhibit low or non-existent coverage on the right side of the heat map.
A different kind of structural lag manifests itself as consistent but relatively low country coverage compared to the “ideal case” scenario. For instance, an indicator may have consistent temporal coverage, but for only 85% or less of possible countries, even though the indicator may be relevant to nearly all countries. Low country coverage may be explained by resource constraints, in that data are more costly to collect in countries where capacity is limited. It may also be that certain economies (for instance, low population or island states) were considered out of scope by data producers. In Figure 3.3 indicators that embody this characteristic exhibit consistent coverage (i.e., uniformly light orange or yellow) along the horizontal axis.
Figure 3.3: Structural Lags
Definition: classification under this explanation was made through consultations with subject matter experts.
In many cases the World Bank is a distributor of indicators it obtains from dozens of external parties. There is some level of “curation” overhead costs to obtain, collate, validate and finalize data from these independent sources, and this overhead may itself affect data availability compared to that of the original sources. This may be especially true where the production cycle of the original indicator varies from that of the Bank’s curation team; for instance, an indicator published monthly by the original source may only be updated quarterly or annually by the World Bank.
Figure 3.4: Curation Lags
Definition: classification under this explanation was made through consultations with the data curation team.
In some cases, the World Bank distributes indicators under legal agreements that affect its availability. The most common cases are licenses that stipulate that the Bank cannot distribute the latest version of a dataset to protect that dataset’s value in cases where providers sell the data commercially. Thus, while more recent data may exist, the Bank is legally prevented from distributing it for a period of time, which can result in an availability gap.
Figure 3.5: Licensing Constraints
Definition: classification under this explanation was made through consultations with the data curation team.
Some indicators are calculated from underlying microdata obtained from household-level or firm-level surveys conducted in country. The survey process constitutes a unique kind of structural lag. Surveys themselves are costly, and are frequently implemented irregularly or according to schedules that vary significantly by country. As a result, data availability may vary not only by time but by country.
Figure 3.6: Survey Dependencies
Definition: classification under this explanation was made through consultations with the data curation team.
Some indicators may not be relevant to or may not be interesting in the context of certain groups of economies. For example, very small economies may not have significant natural resource endowments or particular kinds of economic activity for particular indicators to be meaningful. Indicators designed to measure use of, say, forests, mineral deposits, or levels of trade may be assumed to essentially be zero or “too small to measure.” In a similar vein, some indicators may not be relevant to industrialized economies, such as prevalence of certain diseases (thought to be eradicated), literacy rates, or foreign aid.
Figure 3.7: Limited Relevance
Definition: this explanation was considered for high-income “rich” countries (using the WBG income classification) and small economies, defined as those with populations under 120,000. This population threshold represents a “natural” breakpoint at which population jumps by nearly 40 percent, clearly differentiating economies above and below it. For both “rich” and “small” economies we identified a set of indicators for which at least 80% of economies in the group had no observations at all, as this might suggest a deliberate decision on the part of data producers to exclude the group. From these results we removed indicators that we still considered potentially relevant to rich/small economies despite the lack of available data.
Figure 3.8 shows the results of the explanation framework disaggregated primarily by explanation, and secondarily by primary source (WBG and all external sources).
Figure 3.8: Explanations for data gaps, by source
Overall, the “Structural Lags” explanation is by far the most dominant cause of gaps in ESG data, where it is a contributing factor for 73 of 134 indicators overall, and 40 of 67 indicators in the WBG’s ESG dataset. This frequency likely reflects the prevalence of traditional statistical indicators currently used in ESG, many of which happen to rely on inter-agency cooperation and other intrinsically time consuming modalities to produce comparable data for all countries.
“Survey Dependencies” is the second most significant explanation, contributing to gaps for 24 of 134 indicators overall, and 15 of 67 indicators in the WBG’s dataset. “No Longer Updated” is the third most significant explanation, accounting for 16 of 134 indicators overall, and 6 of 67 indicators in the WBG’s dataset. “Deprecated” is close behind, accounting for 15 of 134 indicators overall, but is not a factor in the WBG’s dataset.
Of the remaining factors, “Licensing” and “Limited Relevance” are the most significant, and in the WBG dataset these are equal to “No Longer Updated” in terms of prevalence. The least significant explanation in both datasets was “Curation Lags,” which accounted for just 2 indicators in both datasets.
Figure 3.9 is similar to Figure 3.8 but disaggregates by explanation and sector.
Figure 3.9: Explanations for data gaps, by sector
This provides a slightly different perspective. Looking at the top 2 explanatory factors, indicators in the social and governance pillars are overly represented within “Structural Lags” and “Survey Dependencies,” suggesting that efforts in these areas would disproportionaately improve these pillars. By comparison, nearly all indicators in the “No Longer Updated” group belong to the environment pillar. The “Licensing” explanation also consists entirely of indicators in the environment pillar.