SDMX Metadata Concepts

Download PDF

0.a. Goal

Goal 16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels

0.b. Target

Target 16.6: Develop effective, accountable and transparent institutions at all levels

0.c. Indicator

Indicator 16.6.2: Proportion of population satisfied with their last experience of public services

0.d. Series

Applies to all series

0.e. Metadata update

2023-03-31

0.g. International organisations(s) responsible for global monitoring

United Nations Development Programme (UNDP)

1.a. Organisation

UNDP Oslo Governance Centre

2.a. Definition and concepts

Definition:

This indicator measures levels of public satisfaction with people’s last experience with public services, in the three service areas of healthcare, education and government services (i.e. services to obtain government-issued identification documents and services for the civil registration of life events such as births, marriages and deaths)[3]. This is a survey-based indicator which emphasizes citizens’ experiences over general perceptions, with an eye on measuring the availability and quality of services as they were actually delivered to survey respondents.

Respondents are asked to reflect on their last experience with each service, and to provide a rating on five ‘attributes’, or service-specific standards, of healthcare, education and government services (such as access, affordability, quality of facilities, etc.). A final question asks respondents for their overall satisfaction level with each service.

It is recommended that survey results, at a minimum, be disaggregated by sex, income and place of residence (urban/rural, administrative regions). To the extent possible, all efforts should be made to also disaggregate results by disability status and by ‘nationally relevant population groups’.

A detailed questionnaire and implementation manual to produce the indicator is defined in the SDG 16 Survey Initiative[4]: The questions for 16.6.2 on healthcare, education and government services can be inserted into existing surveys, using these surveys’ additional batteries on demographics for subsequent disaggregation of results. This modular ‘add-on’ technique also allows for the cross-tabulation of satisfaction levels with other socioeconomic variables found in the larger survey, such as the health conditions of the respondent. This enables a more comprehensive analysis of disparities in the provision of services, and helps to pinpoint specific factors that influence satisfaction levels.

Concepts:

  • Public services: As stated by the United Nations High Commissioner for Human Rights, “States are responsible for delivering a variety of services to their populations, including education, health and social welfare services. The provision of these services is essential to the protection of human rights such as the right to housing, health, education and food. The role of the public sector as service provider or regulator of the private provision of services is crucial for the realization of all human rights, particularly social and economic rights.”[5]

While several definitions of ‘public services’ exist, they tend to have in common a focus on ‘common interest' and on ‘government responsibility’. For instance, the European Commission defines such services as “Services that public authorities of the Member States clarify as being of general interest and, therefore, subject to specific public service obligations.”[6] Similarly, the African Charter on Values and Principles of Public Service and Administration (African Union, 2011) defines a public service as “Any service or public-interest activity that is under the authority of the government administration”.

  • Public services ‘of general interest’: The methodology for SDG 16.6.2 carefully defines the scope of healthcare and education services to ensure that the focus is placed on services that are truly of general interest. In the case of healthcare services, for instance, preventive and primary healthcare services can be said to be truly ‘of general interest’: these services are relevant to everyone and they are most commonly found in both urban and rural areas. This might not be the case for hospitals that provide tertiary care, and as such hospital and specialist care is excluded from the questions on healthcare services. Likewise, in the case of education services, primary and lower secondary education services can be said to be truly ‘of general interest’, given their universality. University education, however, is excluded from the questions on education services.
  • ‘Last experience’ of public services in the past 12 months: Indicator 16.6.2 focuses on respondents’ ‘last experience of public services’, and specifies a reference period of “the past 12 months” to avoid telescoping effects and to minimize memory bias effects. This means that only respondents who will have used healthcare, education and government services in the past 12 months will proceed to answer the survey questions.
  • Service-specific standards – or ‘attributes’: The United Nations High Commissioner for Human Rights explains that “A human rights-based approach to public services is integral to the design, delivery, implementation and monitoring of all public service provision. Firstly, the normative human rights framework provides an important legal yardstick for measuring how well public service is designed and delivered and whether the benefits reach rights-holders”[7]. For instance, the Committee on Economic, Social and Cultural Rights specifies that “The availability, accessibility, acceptability and quality of health-related services should be facilitated and controlled by States. This duty extends to a variety of health-related services ranging from controlling the spread of infectious diseases to ensuring maternal health and adequate facilities for children.”[8] Similarly, with respect to education services, the same Committee underlines that “States should adopt a human rights approach to ensure that [education services are] of an adequate standard and do not exclude any child on the basis of race, religion, geographical location or any other defining characteristic.”[9]
  • Healthcare services: The questions on healthcare services focus on respondents’ experiences (or that of a child in their household who needed treatment and was accompanied by the respondent) with primary healthcare services (over the past 12 months) – that is, basic health care services provided by a government/public health clinic, or covered by a public health system. It can include health care services provided by private institutions, as long as such services are provided at reduced (or no) cost to beneficiaries, under a public health system. Respondents are specifically asked not to include in their answers any experience they might have had with hospital or specialist medical care services (for example, if they had a surgery), or with dental care and teeth exams (because in many countries, dental care is not covered by publicly funded healthcare systems). Attributes-based questions on healthcare services focus on 1) Accessibility (related to geographic proximity, delay in getting appointment, waiting time to see doctor on day of appointment); 2) Affordability; 3) Quality of facilities; 4) Equal treatment for everyone; and 5) Courtesy and treatment (attitude of healthcare staff).
  • Education services: The questions on education services focuses on respondents’ experience with the public school system over the past 12 months, that is, if there are children in their household whose age falls within the age range spanning primary and secondary education in the country. Public schools are defined as “those for which no private tuition fees or major payments must be paid by the parent or guardian of the child who is attending the school; they are state-funded schools.” Respondents are asked to respond separately for primary and secondary schools if children in their household attend school at different levels. Attributes-based questions on education services focus on 1) Accessibility (with a focus on geographic proximity); 2) Affordability; 3) Quality of facilities; 4) Equal treatment for everyone; and 5) Effective delivery of service (Quality of teaching).
  • Government services: The battery on government services focuses exclusively on two types of government services: 1) Services to obtain government-issued identification documents (such as national identity cards, passports, driver’s licenses and voter’s cards) and 2) services for the civil registration of life events such as births, marriages and deaths. This particular focus on these two types of services arises from the high frequency of use of these services. Attributes-based questions on government services focus on 1) Accessibility; 2) Affordability; 3) Equal treatment for everyone; 4) Effective delivery of service (delivery process is simple and easy to understand); and 5) Timeliness.

Selection of relevant disaggregation dimensions

  • Relevant international legal frameworks: Indicator 16.6.2 aims to provide a better understanding of how access to services and the quality of services differ across localities and across various demographic groups. This aim is supported by international human rights law:
  • Article 25 (c) of the International Covenant on Civil and Political Rights provides for the right to equal access to public service. In its report on the role of the public services as an essential component in the promotion and protection of human rights, the United Nations High Commissioner for Human Rights reminds that “States must bear in mind that there are demographic groups in every society that may be disadvantaged in their access to public services, namely women, children, migrants, persons with disabilities, indigenous persons and older persons. States need to ensure that the human rights of these groups are not undermined and that they receive adequate public services.”[10] The High Commissioner also calls attention to the fact that “Poverty acts as a major barrier in relation to public services.”
  • The obligations to ensure equality and non-discrimination are recognized in article 2 of the Universal Declaration of Human Rights and are encountered in many United Nations human rights instruments, such as the International Covenant on Civil and Political Rights (arts. 2 and 26), the International Covenant on Economic, Social and Cultural Rights (art. 2 (2)), the Convention on the Rights of the Child (art. 2), the International Convention on the Protection of the Rights of All Migrant Workers and Members of Their Families (art. 7) and the Convention on the Rights of Persons with Disabilities (art. 5). In terms of public services, this means that States have an immediate obligation to ensure that deliberate, targeted measures are put into place to secure substantive equality and that all individuals have an equal opportunity to enjoy their right to access public services.
  • Empirical analysis: Statistical analysis of available datasets on citizen satisfaction with healthcare and education services[11] shows that the demographic variables that are most strongly correlated with satisfaction with healthcare and education services are (1) income (by far the strongest determinant of satisfaction levels), (2) sex, and (3) place of residence (rural/urban). There is no statistically significant association between the age of respondents and satisfaction levels.
3

The formulation ‘government services’ (also commonly called ‘administrative services’) is used in this metadata to mirror this more colloquial language used in the survey questionnaire. ↑

5

Good Governance Practices for the Protection of Human Rights (United Nations publication, Sales No. E.07.XIV.10), p. 38 – cited in Report of the United Nations High Commissioner for Human Rights on the role of the public service as an essential component of good governance in the promotion and protection of human rights, Human Rights Council, 25th Session, 23 December 2013, A/HRC/25/27 ↑

6

European Commission’s 2011 Communication regarding ‘A Quality Framework for Services of General Interest in Europe’, p. 3 ↑

7

Report of the United Nations High Commissioner for Human Rights on the role of the public service as an essential component of good governance in the promotion and protection of human rights, Human Rights Council, 25th Session, 23 December 2013, A/HRC/25/27 ↑

8

Committee on Economic, Social and Cultural Rights, General Comment No. 14 (2000) on the right to the highest attainable standard of health, para. 4. ↑

9

Committee on Economic, Social and Cultural Rights, general comment No. 13 (1999) on the right to education, para. 1. ↑

10

Report of the United Nations High Commissioner for Human Rights on the role of the public service as an essential component of good governance in the promotion and protection of human rights, Human Rights Council, 25th Session, 23 December 2013, A/HRC/25/27 ↑

11

From the European Social Survey, the European Quality of Life Survey and the Afrobarometer – see more information in the section on “Data Availability”. ↑

2.b. Unit of measure

Percent (%)

2.c. Classifications

Not applicable

3.a. Data sources

This indicator needs to be measured on the basis of data collected by National Statistical Offices (NSOs) through official household surveys.

3.b. Data collection method

NSOs should identify suitable survey vehicles to incorporate the 16.6.2 batteries of question. Some countries may not have an integrated or unified survey covering various public services. In countries where each Ministry/Department/Agency conducts its respective satisfaction survey, the NSO should liaise with each entity to harmonize existing survey questions with this metadata.

3.c. Data collection calendar

To ensure timely capture of changes in levels of citizen satisfaction with public services, NSOs should report data on indicator 16.6.2 at least once every two years. NSOs will need to choose the most appropriate time/period for administering the 16.6.2 batteries of questions. Electoral periods should be avoided, and NSOs should aim for the middle of an electoral term. Experience shows that surveys conducted at the beginning of an electoral term generate more positive responses than surveys conducted at the end of a term.

3.d. Data release calendar

Data will be reported at the international level in the first half of each year.

3.e. Data providers

National Statistical Offices

3.f. Data compilers

United Nations Development Programme (UNDP)

3.g. Institutional mandate

Recent evidence shows that citizens call for responsive and inclusive public institutions with capacity to efficiently deliver services. To advance these aspirations from societies, UNDP helps countries to strengthen responsive and accountable institutions. UNDP recognizes the foundational importance of effective and responsive governance to achieve sustainable development.

4.a. Rationale

Governments have an obligation to provide a wide range of public services that should meet the expectations of their citizens in terms of access, responsiveness and reliability/quality. When citizens cannot afford some essential services, when their geographic or electronic access to services and information is difficult, when the services provided do not respond to their needs and are of poor quality, citizens will naturally tend to report lower satisfaction not only with these services, but also with public institutions and governments. In this regard, it has been shown that citizens’ experience with front-line public services affects their trust in public institutions (OECD 2017, Trust and Public Policy – How Better Governance Can Help Rebuild Public Trust; Eurofound 2018, Societal change and trust in institutions). Mindful of this close connection between service provision/performance, citizen satisfaction and public trust, governments are increasingly interested in better understanding citizens’ needs, experiences and preferences to be able to provide better targeted services, including for underserved populations.

Measuring satisfaction with public services is at the heart of a citizen-centered approach to service delivery and an important outcome indicator of overall government performance. Yet while a large number of countries have experience with measuring citizen satisfaction with public services, there is also large variability in the ways national statistical offices and government agencies in individual countries collect data in this area, in terms of the range of services included, the specific attributes of services examined, question wording and response formats, among other methodological considerations. This variability poses a significant challenge for cross-country comparison of such data.

SDG indicator 16.6.2 aims to generate globally comparable data on satisfaction with public services. To this end, SDG 16.6.2 focuses global reporting on the three service areas of (1) healthcare, (2) education and (3) government services (i.e. services to obtain government-issued identification documents and services for the civil registration of life events such as births, marriages and deaths.)

The rationale for selecting these three public services, (1) healthcare, (2) education and (3) government services, is threefold:

  • First, these are ‘services of consequence’[12], salient for all countries and for both rural and urban populations within countries. They are also among the most common service areas covered by national household or citizen surveys on satisfaction with public services[13].
  • Second, while healthcare and education services are covered by other SDG indicators[14], most of these other indicators rely on administrative sources (i.e. they do not measure people’s direct experiences and level of satisfaction with services) and are mainly focused on measuring the national coverage of a given service.
  • Third, government services are not monitored under other Goals. This is a gap that indicator 16.6.2 can usefully fill, especially since Goal 16 is dedicated to enhancing governance. While Goal 16 does consider birth registration services under indicator 16.9.1, it falls short of measuring satisfaction with the services provided.

With the aim of generating harmonized statistics, indicator 16.6.2 is measured through five attributes-based questions under each service area (e.g. on the accessibility and affordability of the service, the quality of facilities, etc.):

  • The attributes-based questions are asked before the overall satisfaction question. This is based on the intention to enhance the accuracy of the proposed statistical measure on overall satisfaction – that is, to ensure that it correctly reflects the underlying concept that it is intended to capture (based on the specific attributes selected for each service). Experts in governance measurements have found that citizen satisfaction with public services is influenced not only by citizens’ previous experiences with the services, but also by citizens’ expectations[15]. These can be influenced by cultural assumptions about the extent to which service providers should be responsive to citizens’ preferences; by broad public perception of services as communicated through the media; by individual experiences of friends, family and acquaintances; and by how service providers themselves communicate about the type of services they commit to delivering. For instance, national experiences with different question formats have shown that more highly educated respondents who interact more frequently with government (and who possibly have higher awareness of their own rights and of their government’s obligations) have higher expectations in terms of what constitutes a public service of ‘good quality’, compared to the rest of the population[16].
  • Given these multiple influences over citizen expectations of public services, which differ across different national contexts and across different demographic groups, it is essential for this methodology to foster a common understanding among respondents of which aspects of ‘good quality’ service provision are measured. To this end, this methodology ‘primes’ respondents with a common set of attributes of ‘good quality’ service provision prior to asking about their overall satisfaction.
  • National experiences have also shown that asking attributes-based questions prior to an overall satisfaction question helps respondents recall their last experience with more specificity.[17]
  • A key reference used to identify relevant attributes for each service area covered by SDG 16.6.2 is the OECD Serving Citizens Framework (OECD 2015, Government at a Glance), which measures the quality of public services delivered to citizens by assessing three key dimensions of service provision, namely Access[18], Responsiveness[19] and Reliability/Quality[20]. Each one of these three dimensions is then further assessed with specific attributes.
  • The list of attributes in the OECD Serving Citizens Framework is comprehensive and more than a global indicator can feasibly and usefully cover. SDG 16.6.2, therefore, focuses on a limited subset of attributes. The specific set of five attributes used by SDG 16.6.2 to measure satisfaction with healthcare and education service areas was selected on the basis of statistical analysis performed on accessible datasets on satisfaction with these two services, namely from the Afrobarometer and the European Quality of Life Survey. Regression and cluster analysis were conducted on these two datasets to determine the main ‘drivers’ of overall satisfaction among several such attributes, for healthcare and education services[21]. The below table presents the results of this empirical analysis – that is, the subset of five attributes used by SDG 16.6.2 to assess satisfaction in each service area:

Attributes of public services found to be the biggest ‘drivers’ of satisfaction with healthcare and education services (in Europe and Africa)

Attributes

Healthcare service

Education service

1

Accessibility (includes a range of issues such as geographic proximity, delay in getting appointment, waiting time to see doctor on day of appointment)

Accessibility (geographic proximity)

2

Affordability

Affordability

3

Quality of facilities

Quality of facilities

4

Equal treatment for everyone

Equal treatment for everyone

5

Courtesy and treatment (Attitude of healthcare staff)

Effective delivery of service (Quality of teaching)

Source: Statistical analysis by the UNDP Oslo Governance Centre, 2019

  • Attributes-specific questions aim to be specifically informative for national policymaking. The specificity of the information generated by such questions, as well as the focus on citizen experiences rather than simply perceptions, have greater policy use than stand-alone perception data on overall satisfaction, which may not reveal “what needs to be fixed”.
12

While drinking water and sanitation services are also ‘services of consequence’, they are already well covered by SDG indicator 6.1.1 “Proportion of population using safely managed drinking water services” and SDG indicator 6.2.1 “Proportion of population using safely managed sanitation services, including a hand-washing facility with soap and water” which also draw from citizen surveys (Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (JMP) supported by UNICEF and WHO) and look at access, availability and quality. ↑

13

See UNDP Oslo Governance Centre (Nov 2017), A Review of National Statistics Offices’ Practices

and Methodological Considerations in Measuring Citizen Satisfaction with Public Services – Inputs for SDG Indicator 16.6.2 Measurement Methodology ↑

14

For health care services, 3.8.1, 3.5.1, 3.b.1 and 1.4.1, and for education services, 4.a.1 and 4.c.1. ↑

15

See Ellen Lust et al., 2015; Nick Thijs, 2011, Van Ryzin, 2004, for instance. ↑

16

Evidence from Mexico, National Survey of Quality and Governmental Impact (ENCIG) 2017 ↑

17

Ibid. ↑

18

Under the ‘Access’ dimension, three attributes are considered: ‘Affordability’, ‘Geographic proximity’ and ‘Accessibility of information’. ↑

19

Under the ‘Responsiveness’ dimension, three attributes are considered: ‘Citizen-centered approach (courtesy, treatment and integrated services)’, ‘Match of services to special needs’ and ‘Timeliness’. ↑

20

Under the ‘Reliability/Quality’ dimension, three attributes are considered: ‘Effective delivery of services and outcomes’, ‘Consistency in service delivery and outcomes’ and ‘Security/safety’. ↑

21

In the absence of regional or global datasets on satisfaction with government services, the same empirical analysis could not be performed in this service area. To the extent possible, similar attributes are used to assess satisfaction with government services as those used for healthcare and education services, with a distinct focus on the attribute of ‘timeliness’ in the case of government services. ↑

4.b. Comment and limitations

Recommended set of complementary questions to address selection 16.6.2 bias towards ‘users’ of public services

  • Since SDG 16.6.2 refers to people’s ‘last experience’ with public services, the indicator needs to focus on user experiences rather than on non-user perceptions. The experience of users is important, but it is equally important to understand the experiences and perceptions of those who turn elsewhere for services, or who do not access services altogether.
  • For each service area, NSOs are therefore strongly encouraged to administer three complementary questions (see Methodology section) prior to the two ‘priority questions’ to be used for global 16.6.2 reporting. These additional questions will help capture the experience of both users and non-users of public services. They will help identify which population sub-groups who needed healthcare, education and government services did not access the services they needed, and what barriers prevented them from doing so. While the information generated by these additional questions is critical for policymakers to design service provision programmes that ‘leave no one behind’, it is left to the discretion of each country to integrate them or not, as some may already be collecting similar information through existing surveys.

Otherwise, the selection bias inherent in SDG 16.6.2, with its focus on users, can result in mismeasurement due to underlying inequalities in the propensity of various groups to interact with state institutions. In other words, a focus on ‘the last experience with public services’ implicitly means that this indicator includes only those respondents who were privileged enough to access public services in the past year. This means that those (such as ethnic minorities, migrants, the elderly, undocumented workers) who have not been able – or willing – to access the healthcare, education or government services they needed in the past 12 months, often as a consequence of multiple social and economic barriers arising from overlapping forms of marginalization will be undercounted by this indicator. There is a risk therefore that overall satisfaction levels reported on 16.6.2 will over-represent the experience of more privileged groups for whom access to public services is easier, because they have the financial, logistical and intellectual means to do so, and they trust that it is in their interest to do so.

4.c. Method of computation

Reporting on SDG 16.6.2 should be done separately for each of the three service areas. (NB: questions on education may refer to either primary or secondary education – and separate computation of results is recommended for the two levels, resulting in de facto four service areas). Computation involves the computation and reporting of the following three estimates, for each service area:

  1. The share of respondents who responded positively (i.e. ‘strongly agree ‘ or ‘agree’) to each of the five attributes questions;
  2. The simple average of positive responses for the five attribute questions combined; and
  3. The share of respondents who say they are satisfied (i.e. those who responded ‘very satisfied’ or ‘satisfied’) in the overall satisfaction question.

For instance:

Attributes of healthcare services

Positive responses

Attributes of primary education services

Positive responses

Attributes of secondary education services

Positive responses

Attributes of government services

Positive responses

Accessibility

50% respondents 'strongly agree' or 'agree'

Accessibility

Accessibility

Accessibility

Affordability

60% respondents 'strongly agree' or 'agree'

Affordability

Affordability

Affordability

Quality of facilities

73% respondents 'strongly agree' or 'agree'

Quality of facilities

Quality of facilities

Effective service delivery process

Equal treatment for everyone

55% respondents 'strongly agree' or 'agree'

Equal treatment for everyone

Equal treatment for everyone

Equal treatment for everyone

Courtesy and treatment (Attitude of healthcare staff)

42% respondents 'strongly agree' or 'agree'

Effective delivery of service (Quality of teaching)

Effective delivery of service (Quality of teaching)

Timeliness

Average share of positive responses on attributes of healthcare services

(50+60+73+55+42)/5 = 56%

Average share of positive responses on attributes of primary education services

Average share of positive responses on attributes of secondary education services

Average share of positive responses on attributes of government services

Share of respondents satisfied with healthcare services overall

(23% 'very satisfied' + 37% 'satisfied') = 60%

Share of respondents satisfied with primary education services overall

Share of respondents satisfied with secondary education services overall

Share of respondents satisfied with government services overall

*Note: It is important for NSOs to clearly report, for each question, the number of respondents who selected “don’t know” (DK), “not applicable” (NA) or “refuse to answer” (RA), and to exclude such respondents from the calculation of shares of positive responses. For instance, if 65 respondents out of 1000 respondents responded DK, NA or RA on the first attribute-based question, the share of positive responses for this attribute will be calculated out of a total of 935 respondents, and the reporting sheet will indicate that for this particular question, 65 respondents responded DK/NA/RA.

While national-level reporting should cover all three estimates described above, global reporting on SDG indicator 16.6.2 will focus on the last two estimates (i.e. the average share of positive responses across the five attribute questions; and the share of respondents who say they are satisfied in the overall satisfaction question). Additionally, global reporting will also consider the share of positive responses of the five service attributes by the share of people who are satisfied for each of the four service areas (i.e.., primary and secondary education, healthcare, and government services).

Answer scales:

  • To ensure the consistency of measurement in an international context, a standardised approach to response format is required. Available evidence from piloting and other NSO experiences suggests that a four-point Likert-scale with verbal scale anchors is preferable over the alternatives. A four-point scale offers the optimal range of response options for the concepts at hand, in terms of capturing as much meaningful variation between responses as there exists, while remaining understandable for respondents who are not very numerate or literate. Piloting experiences have revealed that offering too few response options (such as a ‘yes/no’ binary response format) would not reveal much variation and might even frustrate some respondents, who might feel their satisfaction level cannot be accurately expressed. Furthermore, the Guidelines on Measuring Subjective Well-Being (OECD, 2013) caution against using “agree/disagree, true/false, and yes/no response formats in the measurement of subjective well-being due to the heightened risk of acquiescence and socially desirable responding”. Meanwhile, piloting experiences have shown that respondents would be equally burdened by too many response categories (such a 7- or 10-point scale), especially if the categories are too close to distinguish between them cognitively.
  • There are different schools of thought on whether an odd or even number of categories is best when using Likert scales. While taking away the middle category forces respondents to voice a positive or negative opinion, and some respondents might find this approach frustrating, several NSOs in developing country contexts favor a Likert scale without a neutral value (such as “neither satisfied nor dissatisfied”). Their preference is motivated by their long-standing survey experience which has shown that when a neutral value is provided, a large proportion (often a majority) of respondents will refrain from expressing their opinion ‘hiding’ behind this middle-point.
  • The survey methodology for 16.6.2 therefore uses a 4-point bipolar Likert scale for all questions (for internal consistency), with the following scale labels: “strongly agree, agree, disagree, strongly disagree” for attributes-based questions, and “very satisfied, satisfied, dissatisfied, very dissatisfied” for overall satisfaction questions. “Don’t know” and “refuse to answer” options are also available, but should not be read out loud, so as to not provide an easy way for respondents to disengage from the subjects of the various questions. When respondents say they “don’t know”, enumerators should repeat the question and simply ask them to provide their best guess. The “don’t know” and “refuse to answer” options should be used only as a last resort.

4.d. Validation

The countries are requested to input the indicators’ data and metadata in a reporting platform following the guidelines in the present metadata sheet. The platform encourages to provide separate information on the survey metadata, namely the source of information for the statistics, the survey instruments, the methodology and protocols and possible. Countries are also requested to insert the statistics on the two questions disaggregated by the pre-specified fields. All inputted information is verified for conformity with the metadata prior to submission.

4.e. Adjustments

Not applicable

4.f. Treatment of missing values (i) at country level and (ii) at regional level

• At country level

There is no treatment of missing values.

• At regional and global levels

There is no imputation of missing values.

4.g. Regional aggregations

Data points will be provided for each region, and globally (i.e. two data points for each service area: combined average % of those who responded positively to the five attributes questions, and % satisfied with the service overall).

4.h. Methods and guidance available to countries for the compilation of the data at the national level

Methods and guidance available to countries for the compilation of data at national level:

See Indicators of Citizen-Centric Public Service Delivery, World Bank (2018)

To disaggregate survey results by disability status, it is recommended that countries use the Short Set of Questions on Disability elaborated by the Washington Group.

Methods and guidance available to countries for the compilation of data at international level:

See Indicators of Citizen-Centric Public Service Delivery, World Bank (2018)

To disaggregate survey results by disability status, it is recommended that countries use the Short Set of Questions on Disability elaborated by the Washington Group.

4.i. Quality management

Statistics for this indicator is inputted in the reporting platform (https://sdg16reporting.undp.org/login). UNDP has dedicated staff to verify the collected data and liaise with the data officers in the agency in the countries.

4.j. Quality assurance

NSOs have the main responsibility to ensure the statistical quality of the data compiled for this indicator. One possible quality assurance mechanism would be to compare results obtained by the NSO with readily available survey results on satisfaction with public services generated by relevant national, regional or global non-official data producers (see potential non-official sources below).

4.k. Quality assessment

UNDP will make available a quality assessment protocol for national statistics office to be used at national level and intended to assess the alignment of data produced with users’ needs, the compliance with guidelines in terms of computations, the timeliness of data production, the accessibility of statistics produced, the consistent use of methodology both in terms of geographic representation and through time, the coherence in terms of data production, and the architecture of data production.

5. Data availability and disaggregation

Data availability:

  • This indicator needs to be measured on the basis of data collected by NSOs through official household surveys.

Description and time series:

There is no existing globally comparable official dataset on the “Proportion of the population satisfied with their last experience of public services.” There is a large variability in the ways NSOs and government agencies in individual countries collect data on citizen satisfaction with public services, in terms of the range of services included, the specific attributes examined, question wording and response formats, etc. This variability poses a significant challenge for cross-country comparability of such data. Several global and regional sources provide comparable data on some measures of citizen satisfaction with public services:

  • The Gallup World Poll surveys people’s satisfaction with local education and healthcare public services in over 150 countries. However, the Gallup World Poll questions do not ask specifically about satisfaction with the last experience of public services, questions do not refer to specific attributes of public services and data is not publicly available.
  • Since launching its first round in 1999/2001, the Afrobarometer[22] has been collecting data biennially on citizens’ satisfaction with healthcare and education services in more than 35 countries in Africa. The Afrobarometer, however, also does not ask about specific attributes of public services and does not ask specifically about satisfaction with the last experience of public services.
  • Starting from 2002, the biennial European Social Survey[23] provides time series data on perception of education and health services in Europe. Once again, these survey questions do not ask specifically about satisfaction with the last experience of public services and do not ask respondents to consider specific attributes of public services when providing their assessment.
  • In its 2016 editions, the European Quality of Life Survey[24] (EQLS) notably introduced questions on specific attributes of service provision in healthcare and education, in additions to questions on overall satisfaction, several of which match the attributes selected for global reporting on 16.6.2. With this focus on the quality of public service provision, this survey could therefore become an appropriate source of data for reporting on SDG 16.6.2 for the 33 participating countries. More specifically, the following corresponding questions in the EQLS have been identified, jointly with Eurofound experts, to report on SDG 16.6.2:

Healthcare services[25]

Attributes

SDG 16.6.2 questions

Corresponding EQLS questions

Access

Q 4.1 It was easy to get to the place where I received medical treatment. (0-3)

Q61 - Thinking about the last time you needed to see or be treated by a GP, family doctor or health centre, to what extent did any of the following make it difficult or not for you to do so? [Very difficult (1); a little difficult (2); not difficult at all (3)]:

a. Distance to GP/doctor’s office / health centre

b. Delay in getting appointment

c. Waiting time to see doctor on day of appointment

Affordability

Q 4.2 Expenses for healthcare services were affordable to you/your household. (0-3)

Q61 – Same as above:

d. Cost of seeing the doctor

Quality of facilities

Q 4.3 The healthcare facilities were clean and in good condition. (0-3)

Q62 - You mentioned that you used GP, family doctor or health centre services. On a scale of 1 to 10 where 1 means very dissatisfied and 10 means very satisfied, tell me how satisfied or dissatisfied you were with each of the following aspects the last time that you used the service.

  1. Quality of the facilities (building, room, equipment)

Equal treatment for everyone

Q 4.4 All people are treated equally in receiving healthcare services in your area. (0-3)

Q63 - To what extent do you agree or disagree with the following about GP, family doctor or health centre services in your area? [on a scale of 1 to 10, where 1 means completely disagree and 10 means completely agree]:

a. All people are treated equally in these services in my area

Courtesy and treatment (Doctor’s attitude)

The doctor or other healthcare staff you saw spent enough time with you [or a child in your household] during the consultation. (0-3)

Q62 - Satisfaction with the following aspects [on a scale of 1 to 10 where 1 means very dissatisfied and 10 means very satisfied]:

c. Personal attention you were given, including staff attitude and time devoted

Overall satisfaction

Overall, how satisfied or dissatisfied were you with the quality of the healthcare services you [or a child in your household] received on that last consultation? (i.e. the last time you [or a child in your household] had a medical examination or treatment in the past 12 months)

Very dissatisfied (0) - Dissatisfied (1) – Satisfied (2) – Very satisfied (3)

Q58 - In general, how would you rate the quality of each of the following public services in [COUNTRY]? [on a scale of one to 10, where 1 means very poor quality and 10 means very high quality]

a. Health services

Education services

Attributes

SDG 16.6.2 questions

Corresponding EQLS questions

Access

Q. 9.1 The school can be reached by public or private transportation, or by walk, in less than 30 minutes and without difficulties. (0-3)

No relevant EQLS question

Affordability

Q. 9.2 School-related expenses (including administrative fees, books, uniforms and transportation) are affordable to you/your household. (0-3)

No relevant EQLS question[26]

Quality of facilities

Q. 9.3 School facilities are in good condition. (0-3)

Q85 - You mentioned that your child or someone in your household attended school. On a scale of 1 to 10 where 1 means very dissatisfied and 10 means very satisfied, please tell me how satisfied or dissatisfied you were with each of the following aspects.

a. Quality of the facilities (building, room, equipment)

Equal treatment for everyone

Q. 9.4 All children are treated equally in the school attended by the child/children in your household. (0-3)

Q86 - To what extent do you agree or disagree with the following statements about school services in your area? Please tell me on a scale of 1 to 10, where 1 means completely disagree and 10 means completely agree.

a. All people are treated equally in these services in my area

Effective delivery of service (Quality of teaching)

Q. 9.5 The quality of teaching is good. (0-3)

Q85 - You mentioned that your child or someone in your household attended school. On a scale of 1 to 10 where 1 means very dissatisfied and 10 means very satisfied, please tell me how satisfied or dissatisfied you were with each of the following aspects.

b. Expertise and professionalism of staff/teachers

e. The curriculum and activities

Overall satisfaction

Q 10. Overall, how satisfied or dissatisfied are you with the quality of education services provided by the primary and/or secondary public schools attended by this child/children in your household?

Are you reporting on:

  1. Primary school in your area ___
  2. Secondary school in your area ___

Very dissatisfied (0) - Dissatisfied (1) – Satisfied (2) – Very satisfied (3)

Q58 - In general, how would you rate the quality of each of the following public services in [COUNTRY]? [on a scale of one to 10, where one means very poor quality and 10 means very high quality]

b. Education system

Disaggregation categories

Indicator 16.6.2 aims to measure how access to services and how the quality of services differs across various demographic groups. Empirical analysis to identify the strongest demographic determinants of citizen satisfaction with public services reveals that the most relevant disaggregation categories for SDG indicator 16.6.2 are (1) income, (2) sex and (3) place of residence (urban/rural, and by administrative region e.g., by province, state, district, etc.)

At a minimum, results for each one of the three service areas covered by this indicator (healthcare, education and government services) should be disaggregated by these three variables:

  • Income: Income (or expenditure) quintiles
  • Sex: Male/Female
  • Place of residence: Living in urban/rural areas and/or living in which administrative region (province, state, district, etc.)[27]

To the extent possible, all efforts should be made to also disaggregate results by disability status and by ‘nationally relevant population groups’:

  • Disability status: ‘Disability’ is an umbrella term covering long-term physical, mental, intellectual or sensory impairments which in interaction with various barriers may hinder the full and effective participation of disabled persons in society on an equal basis with others[28]. If possible, NSOs are encouraged to add the Short Set of Questions on Disability developed by the Washington Group to the survey vehicle used to administer the 16.6.2 batteries to disaggregate results by disability status.
  • Nationally relevant population groups: groups with a distinct ethnicity, language, religion, indigenous status, nationality or other characteristics.[29]
  • Age: Empirical analysis shows that there is no statistically significant association between the age of respondents and satisfaction levels. However, if countries choose to also disaggregate results by age, it is recommended to follow UN standards for the production of age-disaggregated national population statistics, using the following age groups: (1) below 25 years old, (2) 25-34, (3) 35-44, (4) 45-54, (5) 55-64 and (6) 65 years old and above.
22

The Afrobarometer is conducting its public attitude surveys on democracy, governance, economic conditions, and related issues in more than 35 countries in Africa. ↑

23

In total, 37 countries have taken part in at least one round of the ESS since its inception. Surveys are conducted by leading academics and social research professionals. ↑

24

EQLS 2016 – the fourth survey in the series – covered the 28 EU Member States and 5 candidate countries (Albania, the former Yugoslav Republic of Macedonia, Montenegro, Serbia and Turkey). ↑

25

Note: For healthcare services, EQLS data would allow for the separate reporting of results (across all questions) on (1) primary care services (GP / doctor’s office / health centre) and (2) hospital or medical specialist services. Separate reporting on these two types of health care would be particularly relevant for the ‘affordability’ attribute, given in European countries, primary care services typically cost little; more relevant would be to assess the affordability of hospital or medical specialist services, using question 67.e. ↑

26

However, question HC100 on ‘Affordability of formal education’ could be used in the European Union Statistics on Income and Living Conditions (EU-SILC) ad hoc module 2016. ↑

27

Based on the premise that decentralization efforts are aimed at extending local rights and responsibilities across the national territory, indicator 16.6.2 can help detect unequal access to services and disparities in the quality of services across localities. There is a risk for erroneous conclusions to be drawn from national aggregates unable to detect variations at sub-national level. ↑

28

UN General Assembly, Convention on the Rights of Persons with Disabilities: resolution / adopted by the General Assembly, 24 January 2007, A/RES/61/106, available at: http://www.refworld.org/docid/45f973632.html ↑

29

The population of a country is a mosaic of different population groups that can be identified according to racial, ethnic, language, indigenous or migration status, religious affiliation, or sexual orientation, amongst other characteristics. For the purpose of this indicator, particular focus is placed on minorities. Minority groups are groups that are numerically inferior to the rest of the population of a state, in a non-dominant position, whose members—being nationals of the state—possess ethnic, religious or linguistic characteristics differing from those of the rest of the population and show, even if only implicitly, a sense of solidarity directed towards preserving their culture, traditions, religion or language. While the nationality criterion included in the above definition has often been challenged, the requirement to be in a non-dominant position remains important (OHCHR, 2010). Collecting survey data disaggregated by population groups should be subject to the legality of compiling such data in a particular national context and to a careful assessment of the potential risks of collecting such data for the safety of respondents. ↑

6. Comparability/deviation from international standards

Sources of discrepancies:

There is no internationally estimated data for this indicator.

7. References and Documentation