Skip to main content

3. Measurement of utility

Question: How is the utility of synthetic data measured?

Rationale​

A synthetic file is useful when the things users will do with it give answers close to what the real file gives. That is measured at three levels. Marginals: does each variable have the same distribution. Associations: do pairs of variables relate as in the real file. Analyses: do the statistics and models users will compute come out the same. The first two are general and cheap; the third is specific to the uses and is the one that decides. A file can pass the first two and fail the third, and the usual failure of a method that ignores relationships is visible in all three, as the running example shows.

Target state​

  • Every synthetic file has a utility report with marginal, association, and target-analysis measures against the real file, produced on the organization's infrastructure.
  • The target analyses are the ones the intended uses require, listed in the purpose statement, and the report says which hold and which do not.
  • The decision to release is taken from the report against thresholds the methodologist set, and the report is attached to the synthesis record.

Maturity levels​

LevelSteps
FoundationalRun the utility report; compare marginals and the two example analyses.
AI-readyAdd the target analyses of the intended uses; set thresholds; compare methods on the same report; attach the report to the record.
AI-nativeRun the report as a pipeline stage at each synthesis; tune the method by the target analyses; publish the report with the file.
Running example

The crude synthesis of chapter 2 matches every marginal within a few percent and loses the association between sex and labour force status: Cramér's V falls from 0.13 to 0.05, and the employment rate for men falls from 0.56 to 0.40 while the rate for women barely moves. A user studying the gender employment gap would get the wrong answer from this file, and the report says so before release.

Recipes​

Recipe

3.1 Run the utility report at three levels

Level
Foundational
Skills
Python
Indicative time
Minutes per file

Result: the organization has a report that compares the synthetic file with the real one at three levels: each variable's distribution, the relationships between pairs of variables, and the statistics users will compute.

The script reads the real and the synthetic file, which have the same columns, and reports three things. For each variable, how different its distribution is: for a categorical variable the total variation distance (0 means identical, 1 means no overlap), for a numeric variable the shift in the median and in the spread. For each pair of variables, how different their association is: Cramér's V for two categorical variables (0 means unrelated, 1 means fully determined) and the correlation for two numeric ones. And for two target statistics, the employment rate by sex and the median income by education, the value in each file side by side.

"""Compare a synthetic table with the real one on marginals, associations,
and a target analysis.

Reads the real and the synthetic CSV (same columns) and reports:

    marginals      for each categorical column the total variation
                   distance between the two distributions; for each
                   numeric column the difference of the medians and of
                   the interquartile ranges relative to the real ones
    associations   for each pair of categorical columns the difference
                   in Cramér's V; for numeric pairs the difference in
                   Pearson correlation
    target         a statistic users compute (here the employment rate
                   by sex and the median income by education) in both
                   files side by side

Columns with at most 12 distinct values are treated as categorical.
Empty values are their own category for categorical columns and are
dropped for numeric ones. Uses pandas and scipy (Cramér's V).

Usage:
    python utility_report.py real_sample.csv synthetic_sample.csv

What this does not do: these are general-purpose measures. The utility
that matters is whether the analyses users will run give the same
answers, which the target section shows for two statistics; the
organization adds the analyses its users run. A synthetic file that
scores well on marginals and badly on associations is the usual
failure of independent synthesis, which the running example shows.
"""

from __future__ import annotations

import argparse
import sys
from itertools import combinations

import pandas as pd
from scipy.stats.contingency import association

CATEGORICAL_MAX = 12


def kinds(df: pd.DataFrame) -> dict[str, str]:
    """Categorical when few distinct values or non-numeric; numeric otherwise."""
    out = {}
    for c in df.columns:
        present = df[c][df[c] != ""]
        numeric = (
            len(present) > 0 and pd.to_numeric(present, errors="coerce").notna().all()
        )
        out[c] = (
            "categorical"
            if df[c].nunique() <= CATEGORICAL_MAX or not numeric
            else "numeric"
        )
    return out


def tvd(a, b) -> float:
    """Total variation distance between two categorical distributions (0 identical, 1 disjoint)."""
    pa, pb = (
        pd.Series(list(a)).value_counts(normalize=True),
        pd.Series(list(b)).value_counts(normalize=True),
    )
    return 0.5 * (pa.subtract(pb, fill_value=0).abs().sum())


def cramers_v(x: pd.Series, y: pd.Series) -> float:
    table = pd.crosstab(x, y)
    return (
        association(table.to_numpy(), method="cramer") if min(table.shape) > 1 else 0.0
    )


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("real")
    parser.add_argument("synthetic")
    args = parser.parse_args(argv)
    real, synth = (
        pd.read_csv(p, dtype=str, keep_default_na=False)
        for p in (args.real, args.synthetic)
    )
    kind = kinds(real)
    cats = [c for c, k in kind.items() if k == "categorical"]
    nums = [c for c, k in kind.items() if k == "numeric"]
    num = {
        name: df[nums].apply(pd.to_numeric, errors="coerce")
        for name, df in (("real", real), ("synthetic", synth))
    }
    print(f"real {len(real)} rows, synthetic {len(synth)} rows, {len(kind)} columns\n")

    print("marginals")
    for c in kind:
        if kind[c] == "categorical":
            print(f"  {c:<12} categorical  TVD {tvd(real[c], synth[c]):.3f}")
        else:
            qa, qb = (
                num["real"][c].quantile([0.25, 0.5, 0.75]),
                num["synthetic"][c].quantile([0.25, 0.5, 0.75]),
            )
            med = (qb[0.5] - qa[0.5]) / qa[0.5] if qa[0.5] else 0.0
            iqr_a, iqr_b = qa[0.75] - qa[0.25], qb[0.75] - qb[0.25]
            print(
                f"  {c:<12} numeric      median diff {med:+.1%}, IQR diff {(iqr_b - iqr_a) / iqr_a if iqr_a else 0:+.1%}"
            )

    print("\nassociations (real vs synthetic)")
    for a, b in combinations(cats, 2):
        va, vb = cramers_v(real[a], real[b]), cramers_v(synth[a], synth[b])
        print(
            f"  {a} x {b:<12} Cramer's V {va:.2f} vs {vb:.2f}{'  <- association lost' if va - vb > 0.1 else ''}"
        )
    for a, b in combinations(nums, 2):
        ra, rb = (
            num["real"][[a, b]].corr().iloc[0, 1],
            num["synthetic"][[a, b]].corr().iloc[0, 1],
        )
        print(f"  {a} x {b:<12} correlation {ra:+.2f} vs {rb:+.2f}")

    print("\ntarget analyses")
    for sex in ("1", "2"):
        for name, df, numbers in (
            ("real", real, num["real"]),
            ("synthetic", synth, num["synthetic"]),
        ):
            group = df[(df["sex"] == sex) & numbers["age"].between(15, 64)]
            rate = (group["lfs_status"] == "1").mean() if len(group) else 0.0
            print(
                f"  employment rate, sex {sex}, ages 15-64: {name:<9} {rate:.3f} (n={len(group)})"
            )
    for educ in ("0", "1", "2", "3"):
        medians = {
            name: numbers.loc[df["educ"] == educ, "income"].median()
            for name, df, numbers in (
                ("real", real, num["real"]),
                ("synthetic", synth, num["synthetic"]),
            )
        }
        print(
            f"  median income, education {educ}: real {medians['real']:.0f}, synthetic {medians['synthetic']:.0f}"
        )
    return 0


if __name__ == "__main__":
    sys.exit(main())
python utility_report.py real_sample.csv synthetic_sample.csv

Output for the running example:

real 300 rows, synthetic 300 rows, 6 columns

marginals
  region       categorical  TVD 0.007
  sex          categorical  TVD 0.020
  age          numeric      median diff -4.3%, IQR diff +3.0%
  educ         categorical  TVD 0.053
  lfs_status   categorical  TVD 0.007
  income       numeric      median diff -20.6%, IQR diff -9.1%

associations (real vs synthetic)
  region x sex          Cramer's V 0.08 vs 0.11
  region x educ         Cramer's V 0.12 vs 0.16
  region x lfs_status   Cramer's V 0.07 vs 0.08
  sex x educ         Cramer's V 0.13 vs 0.06
  sex x lfs_status   Cramer's V 0.13 vs 0.05
  educ x lfs_status   Cramer's V 0.13 vs 0.15
  age x income       correlation -0.06 vs +0.01

target analyses
  employment rate, sex 1, ages 15-64: real      0.556 (n=124)
  employment rate, sex 1, ages 15-64: synthetic 0.397 (n=116)
  employment rate, sex 2, ages 15-64: real      0.473 (n=110)
  employment rate, sex 2, ages 15-64: synthetic 0.455 (n=123)
  median income, education 0: real 325, synthetic 300
  median income, education 1: real 625, synthetic 600
  median income, education 2: real 1125, synthetic 1075
  median income, education 3: real 1900, synthetic 2400

The marginals block gives one line per variable with its distance measure. The associations block gives, for each pair, the measure in the real file and in the synthetic one side by side. The target analyses block gives each statistic in both files with the number of records it rests on.

  1. Run the report where the real file is, on the organization's own infrastructure. Only the report leaves.
  2. Read the distributions. A total variation distance above a few percent on a categorical variable, or a median shifted by more than a few percent, marks a variable to look at.
  3. Read the associations. A pair whose Cramér's V or correlation fell substantially is a relationship the method lost. In the example, the link between sex and labour force status fell from 0.13 to 0.05.
  4. Read the target statistics. These are what users will compute, and the gap is how wrong they will be. In the example, the employment rate for men is 0.56 in the real file and 0.40 in the synthetic one.

Worked example. The methodologist reads the report above. The marginals are close, except that the median income is 20 percent lower in the synthetic file, which the crude method's conditioning on education alone explains. Among the associations, two pairs stand out: sex and labour force status fell from 0.13 to 0.05, and sex and education from 0.13 to 0.06, because the crude method drew sex independently of everything else. The target analyses show the consequence: the employment rate for men is 16 points too low, while the rate for women is close. The methodologist writes "fails on the gender employment gap; method ignores relationships with sex" in the report and sends the file back to chapter 2, where the sequential CART synthesis with sex early in the visit sequence restores the association to 0.12.

What this does not do. The report's two target statistics are examples; recipe 3.2 adds the organization's own. Three hundred records give rough figures; the real file's size gives the precision.

Recipe

3.2 Add the target analyses of the intended uses

Level
AI-ready
Skills
Methodologist; subject-matter analyst
Indicative time
Half a day per file

Result: the report includes the statistics and models that the intended uses require, computed on both files, with the differences and the real file's sampling error beside them.

  1. From the purpose statement's intended uses, list the analyses: the published tables users reproduce to test their code, the regressions a course teaches, the indicators a method study needs.
  2. Compute each on the real and the synthetic file (with weights where the survey is weighted). Report the difference, and beside it the real file's sampling error for the same statistic. A difference within the sampling error is the target.
  3. Where a model is fitted (a regression), compare the coefficients and their confidence intervals; the overlap of the intervals is a standard measure of how close the two files are.
  4. Add the analyses to the report script, so that every future synthesis of the survey reports them.

Worked example. The purpose statement's uses give four target analyses for the labour force file. They are the published table of unemployment by sex and age group, the employment rate by region, the median hours by occupation major group, and the course's regression of log income on education, sex, age, and region. On the full file the sequential synthesis reproduces the unemployment table within 0.8 points in every cell, where the real file's sampling error is about 1.2 points, and the regional employment rates within 1.5 points. The regression's coefficient on education is 0.21 in the real file and 0.19 in the synthetic one, with confidence intervals that overlap by 80 percent. The four analyses are added to the script, and the report lists them with the sampling errors beside the differences.

What this does not do. The analyses cover what the organization anticipated. A user's new analysis may depend on a relationship the method weakened, which is why the synthesis record lists what was measured and tells users to treat everything else as unverified.

Recipe

3.3 Set thresholds and decide

Level
AI-ready
Skills
Methodologist; head of the microdata unit
Indicative time
An hour per file

Result: the release decision has been taken by comparing the report with thresholds set for the file's intended uses, and it is recorded in the synthesis record.

  1. Set thresholds at each level for the file's uses. For testing code, the variable types and distributions matter most. For teaching, the main relationships and the target analyses matter. For method development, the specific structures the method needs matter.
  2. Compare the report with the thresholds. A file that fails goes back to chapter 2 with the failing measures as the thing to improve, or its intended uses are narrowed.
  3. Record the decision, the thresholds, and the report in the synthesis record (chapter 6).

Worked example. For the labour force file the methodologist sets three thresholds: a total variation distance under 0.05 on every categorical variable, no association whose Cramér's V changes by more than 0.05, and every target analysis within the real file's sampling error. The sequential synthesis passes the first and the third and fails the second on one pair, occupation and industry, where the measure falls from 0.42 to 0.33. The intended uses do not depend on that pair, so the decision is to release with the limitation stated in the synthesis record: "the joint distribution of occupation and industry is weaker than in the real file; analyses of that pair are unverified". The transformer-based alternative, which keeps the pair, is noted as the method for the next synthesis.

What this does not do. Thresholds encode the organization's judgement of "close enough" for a use. No threshold in this guide makes a synthetic file suitable for estimating statistics.

Implementation options​

  • Measures: total variation distance and quantile differences for marginals; Cramér's V and correlation differences for associations; target statistics with sampling error, model coefficients with interval overlap.
  • Tools: the script in this chapter; synthpop's compare() and utility.gen(); SDMetrics quality reports; the organization's own analysis code run on both files.
  • Thresholds: per level and per use, set by the methodologist.

World Bank examples​

  • The REaLTabFormer paper evaluates synthetic tables by machine-learning efficacy and statistical similarity, the association and target levels of this chapter.
  • The program's Anomaly Detection and Explanation work compares distributions across series, the same marginal comparisons applied to synthetic files.

Resources​

Common mistakes​

  • Marginals only. Every crude method passes them.
  • Target analyses nobody uses. The intended uses name them.
  • Utility measured on the real file's server by someone who then shares the real file's numbers. The report is aggregate and reviewed before it leaves.
  • A failed threshold explained away. The file goes back or the uses narrow.

Verification​

  • Report. A report exists for every synthetic file at the three levels, with the target analyses of its uses.
  • Thresholds. The thresholds are set per use and recorded, and the decision follows them.
  • Record. The report is attached to the synthesis record.

Checklist​

0 of 6 done

Ticks are saved in this browser only.