Skip to main content

3. Croissant records for loading

Question: How is a dataset described so that tools can load it?

Rationale​

A catalog record tells a person what a dataset is. A loading record tells a program how to read it: which files, in which format, with which checksum; which fields each file holds, of which type; which field is the record's key; and which records belong to which split. Croissant is the format the machine-learning community has agreed on for this. It is a JSON-LD document (JSON with a vocabulary, so that each key has a defined meaning) built on schema.org, which means that a Croissant record is also a schema.org Dataset record that search engines read. Dataset repositories such as Hugging Face and Kaggle publish a Croissant record for every dataset, and loading libraries read one to fetch and parse the files.

For a statistical organization, Croissant is the bridge between the catalog and the training script. The catalog record stays the authority for what the dataset is; the Croissant record, generated from the same dictionary, says how to load it. A record has four layers: the dataset's metadata (name, description, licence, URL, version, citation), its resources (the files, each with a checksum), its structure (record sets and their fields), and machine-learning semantics (splits, labels). This chapter generates the first three layers and the splits from the files of chapter 2 and chapter 6, and checks the result.

Target state​

  • Every published dataset has a Croissant record beside its files, generated from the dictionary, with a checksum for every file and a typed field for every column.
  • The record names the splits, so that a loader can fetch the training part and leave the test part alone.
  • The record is checked on every release, and validated with the reference validator before publication.

Maturity levels​

LevelSteps
FoundationalGenerate a Croissant record for one dataset from its dictionary and check it.
AI-readyAdd the splits to the record, validate with mlcroissant, and publish the record on the dataset page and in the catalog.
AI-nativeGenerate and validate the record as a pipeline stage at each release; register datasets in a public repository that reads Croissant.
Running example

The occupation-coding dataset's record names two files (the data and the splits, each with its SHA-256), a record set of twelve typed fields generated from the dictionary, a record set that enumerates the three split names, and a record set that joins each record to its split.

Recipes​

Recipe

3.1 Generate the record from the dictionary

Level
Foundational
Skills
Python
Indicative time
Minutes

Result: the dataset has a Croissant record, written from the data file, the dictionary, and the dataset's metadata, with a checksum for each file and a typed field for each column.

The script reads the data file and the dictionary and takes the dataset-level metadata as arguments. It writes the file list with SHA-256 checksums (a fingerprint of the file's content that changes if one byte changes), one field per dictionary row with its type, and, when the splits file is given, the split record sets:

"""Write a Croissant record for a CSV dataset from its dictionary.

Reads a data file (CSV) and its dictionary (name, label, type,
description, role) and writes a Croissant 1.0 JSON-LD record: the
dataset's name, description, licence, URL, version, and citation; the
file as a FileObject with its SHA-256; and one RecordSet with a Field per
column, typed from the dictionary. With --splits, adds the split
assignment file as a second FileObject and a RecordSet that joins it to
the records. Standard library only.

Usage:
    python build_croissant.py lfs_occupation_ml.csv dictionary.csv \
        --name "LFS occupation coding (example)" \
        --description "Job descriptions with ISCO-08 codes from the labour force survey." \
        --url https://stats.example/datasets/lfs-occupation-ml \
        --license https://creativecommons.org/licenses/by/4.0/ \
        --version 1.0.0 --date-published 2026-10-09 --cite-as "National Statistics Office (example). 2026. ..." \
        --splits splits.csv -o lfs_occupation_ml.croissant.json

What this does not do: it writes the structural layers (metadata, files,
fields, splits). It does not validate the record against the full
specification (mlcroissant does), and the description, licence, and
citation are only as right as the arguments given.
"""

from __future__ import annotations

import argparse
import csv
import hashlib
import json
import sys
from pathlib import Path

CONTEXT = {
    "@language": "en",
    "@vocab": "https://schema.org/",
    "citeAs": "cr:citeAs",
    "column": "cr:column",
    "conformsTo": "dct:conformsTo",
    "cr": "http://mlcommons.org/croissant/",
    "rai": "http://mlcommons.org/croissant/RAI/",
    "data": {"@id": "cr:data", "@type": "@json"},
    "dataType": {"@id": "cr:dataType", "@type": "@vocab"},
    "dct": "http://purl.org/dc/terms/",
    "equivalentProperty": "cr:equivalentProperty",
    "examples": {"@id": "cr:examples", "@type": "@json"},
    "extract": "cr:extract",
    "field": "cr:field",
    "fileProperty": "cr:fileProperty",
    "fileObject": "cr:fileObject",
    "fileSet": "cr:fileSet",
    "format": "cr:format",
    "includes": "cr:includes",
    "isLiveDataset": "cr:isLiveDataset",
    "jsonPath": "cr:jsonPath",
    "key": "cr:key",
    "md5": "cr:md5",
    "parentField": "cr:parentField",
    "path": "cr:path",
    "recordSet": "cr:recordSet",
    "references": "cr:references",
    "regex": "cr:regex",
    "repeated": "cr:repeated",
    "replace": "cr:replace",
    "samplingRate": "cr:samplingRate",
    "sc": "https://schema.org/",
    "separator": "cr:separator",
    "source": "cr:source",
    "subField": "cr:subField",
    "transform": "cr:transform",
}

TYPES = {
    "numeric": "sc:Float",
    "integer": "sc:Integer",
    "categorical": "sc:Text",
    "text": "sc:Text",
    "id": "sc:Text",
    "boolean": "sc:Boolean",
}


def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


def file_object(path: Path, ident: str, description: str) -> dict:
    return {
        "@type": "cr:FileObject",
        "@id": ident,
        "name": path.name,
        "description": description,
        "contentUrl": path.name,
        "encodingFormat": "text/csv",
        "sha256": sha256(path),
    }


def field(
    record_set: str, name: str, description: str, data_type: str, file_id: str
) -> dict:
    return {
        "@type": "cr:Field",
        "@id": f"{record_set}/{name}",
        "name": f"{record_set}/{name}",
        "description": description,
        "dataType": data_type,
        "source": {"fileObject": {"@id": file_id}, "extract": {"column": name}},
    }


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("data")
    parser.add_argument("dictionary")
    parser.add_argument("--name", required=True)
    parser.add_argument("--description", required=True)
    parser.add_argument("--url", required=True)
    parser.add_argument("--license", required=True, help="a URL")
    parser.add_argument("--version", required=True)
    parser.add_argument("--cite-as", required=True)
    parser.add_argument("--creator", default="National Statistics Office (example)")
    parser.add_argument(
        "--date-published", required=True, help="ISO date of the release"
    )
    parser.add_argument(
        "--splits", help="CSV with the record identifier and a split column"
    )
    parser.add_argument("-o", "--output", default="dataset.croissant.json")
    args = parser.parse_args(argv)

    data = Path(args.data)
    with open(args.dictionary, newline="", encoding="utf-8") as f:
        dictionary = list(csv.DictReader(f))
    with open(data, newline="", encoding="utf-8") as f:
        header = next(csv.reader(f))
    names = {d["name"] for d in dictionary}
    missing = [c for c in header if c not in names]
    if missing:
        print(
            f"columns without a dictionary row: {', '.join(missing)}", file=sys.stderr
        )
        return 1

    records = "records"
    fields = []
    for d in dictionary:
        desc = d["description"] + (f" Role: {d['role']}." if d.get("role") else "")
        fields.append(
            field(
                records, d["name"], desc, TYPES.get(d["type"], "sc:Text"), "data-file"
            )
        )
    record_sets = [
        {
            "@type": "cr:RecordSet",
            "@id": records,
            "name": records,
            "description": f"One record per row of {data.name}.",
            "key": {"@id": f"{records}/{header[0]}"},
            "field": fields,
        }
    ]
    distribution = [file_object(data, "data-file", "The data, one record per row.")]

    if args.splits:
        splits = Path(args.splits)
        with open(splits, newline="", encoding="utf-8") as f:
            split_names = sorted({r["split"] for r in csv.DictReader(f)})
        distribution.append(
            file_object(
                splits, "splits-file", "The split assignment, one row per record."
            )
        )
        record_sets.append(
            {
                "@type": "cr:RecordSet",
                "@id": "split_names",
                "name": "split_names",
                "description": "The named splits of the dataset.",
                "dataType": "cr:Split",
                "key": {"@id": "split_names/name"},
                "field": [
                    {
                        "@type": "cr:Field",
                        "@id": "split_names/name",
                        "name": "name",
                        "description": "The split name.",
                        "dataType": "sc:Text",
                    }
                ],
                "data": [{"split_names/name": s} for s in split_names],
            }
        )
        record_sets.append(
            {
                "@type": "cr:RecordSet",
                "@id": "record_splits",
                "name": "record_splits",
                "description": "The split of each record, joined on the record identifier.",
                "field": [
                    {
                        "@type": "cr:Field",
                        "@id": "record_splits/record",
                        "name": "record_splits/record",
                        "description": "The record identifier.",
                        "dataType": "sc:Text",
                        "source": {
                            "fileObject": {"@id": "splits-file"},
                            "extract": {"column": header[0]},
                        },
                        "references": {"field": {"@id": f"{records}/{header[0]}"}},
                    },
                    {
                        "@type": "cr:Field",
                        "@id": "record_splits/split",
                        "name": "record_splits/split",
                        "description": "The split name.",
                        "dataType": "sc:Text",
                        "source": {
                            "fileObject": {"@id": "splits-file"},
                            "extract": {"column": "split"},
                        },
                        "references": {"field": {"@id": "split_names/name"}},
                    },
                ],
            }
        )

    record = {
        "@context": CONTEXT,
        "@type": "sc:Dataset",
        "conformsTo": "http://mlcommons.org/croissant/1.0",
        "name": args.name,
        "description": args.description,
        "url": args.url,
        "license": args.license,
        "version": args.version,
        "datePublished": args.date_published,
        "citeAs": args.cite_as,
        "creator": {"@type": "Organization", "name": args.creator},
        "distribution": distribution,
        "recordSet": record_sets,
    }
    Path(args.output).write_text(
        json.dumps(record, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
    )
    print(
        f"{Path(args.output).name}: {len(distribution)} file(s), {len(record_sets)} record set(s), {len(fields)} field(s)"
    )
    return 0


if __name__ == "__main__":
    sys.exit(main())
python build_croissant.py lfs_occupation_ml.csv dictionary.csv \
    --name "LFS occupation coding (example)" \
    --description "Job descriptions from the labour force survey with ISCO-08 codes assigned by trained coders, for training and evaluating automatic occupation coders." \
    --url https://stats.example/datasets/lfs-occupation-ml \
    --license https://creativecommons.org/licenses/by/4.0/ \
    --version 1.0.0 --date-published 2026-10-09 \
    --cite-as "National Statistics Office (example). 2026. LFS occupation coding (example), version 1.0.0. https://stats.example/datasets/lfs-occupation-ml" \
    --splits splits.csv -o lfs_occupation_ml.croissant.json

Output for the running example:

lfs_occupation_ml.croissant.json: 2 file(s), 3 record set(s), 12 field(s)

The start of the record it writes, with the four dataset-level parts that every record carries:

{
  "@context": { "...": "the Croissant vocabulary" },
  "@type": "sc:Dataset",
  "conformsTo": "http://mlcommons.org/croissant/1.0",
  "name": "LFS occupation coding (example)",
  "description": "Job descriptions from the labour force survey with ISCO-08 codes ...",
  "url": "https://stats.example/datasets/lfs-occupation-ml",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "version": "1.0.0",
  "datePublished": "2026-10-09",
  "citeAs": "National Statistics Office (example). 2026. ...",
  "distribution": [
    {
      "@type": "cr:FileObject",
      "@id": "data-file",
      "name": "lfs_occupation_ml.csv",
      "contentUrl": "lfs_occupation_ml.csv",
      "encodingFormat": "text/csv",
      "sha256": "e6b0b1fd..."
    }
  ],
  "recordSet": [ { "@id": "records", "field": [ "... one per column ..." ] } ]
}
  1. Run the script with the dataset's metadata. The name, URL, licence, version, and citation are the same as in the card and the catalog record.
  2. Read the record once. Each field's description comes from the dictionary, with its role appended, so that a loader's user sees which field is the label.
  3. Keep the record in the release folder. It is regenerated whenever the data, the dictionary, or the splits change, because the checksums change.

Worked example. The command above writes the record for the occupation dataset. Its distribution lists the data file and the splits file with their SHA-256 digests. Its first record set, records, has twelve fields, one per dictionary row. The field for isco_code has the type sc:Text and a source that names the data file and the column. Its description reads "Four-digit ISCO-08 code assigned by a trained coder; 9999 means not codable (role: label)". Its second record set enumerates the three split names, and its third joins each record_id to its split through the splits file. A partner's loading script reads the record, fetches the two files, checks the digests, and asks for the training part by name.

What this does not do. The script writes the structural layers for a CSV dataset with one record set. Datasets with several files that join (images and their annotations), nested fields, or data types beyond text and numbers (bounding boxes, image references) need fields the script does not write; the Croissant specification and the mlcroissant examples show them.

Recipe

3.2 Check the record and validate it

Level
Foundational
Skills
Python
Indicative time
Minutes

Result: a check runs on every release and confirms that the record has every part a loader needs, that every field's source names a file in the record, and that the checksums match the files on disk.

The script reads the record and the files beside it:

"""Check a Croissant record for the parts a loader and a catalog need.

Reads a Croissant JSON-LD file and checks: the dataset type and the
conformsTo declaration; the presence of name, description, url, license,
version, and citeAs; that every file in the distribution has a
contentUrl, an encodingFormat, and a sha256; that every record set has
fields, that every field has a dataType, and that every field source
names a file that exists in the distribution; and, when the files are
present beside the record, that each file's SHA-256 matches. Exit status
1 on any error. Uses mlcroissant, the reference implementation, for the specification check.

Usage:
    python croissant_check.py lfs_occupation_ml.croissant.json

What this does not do: it checks structure and integrity. It does not
run the full Croissant validator (mlcroissant), does not load the data,
and does not judge whether the description or the licence are right.
"""

from __future__ import annotations

import argparse
import hashlib
import json
import sys
from pathlib import Path

import mlcroissant as mlc

REQUIRED = [
    "name",
    "description",
    "url",
    "license",
    "version",
    "datePublished",
    "citeAs",
]


def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("record")
    parser.add_argument(
        "--files-dir",
        help="where the files named by contentUrl are; default: beside the record",
    )
    args = parser.parse_args(argv)
    path = Path(args.record)
    rec = json.loads(path.read_text(encoding="utf-8"))
    files_dir = Path(args.files_dir) if args.files_dir else path.parent
    errors: list[str] = []
    warnings: list[str] = []

    # 1. the specification, through the reference implementation
    try:
        dataset = mlc.Dataset(jsonld=str(path))
        warnings += (
            [
                w.strip()
                for w in str(dataset.metadata.issues.warnings or "").split("\n")
                if w.strip() and "datePublished" not in w
            ]
            if hasattr(dataset.metadata, "issues")
            else []
        )
    except mlc.ValidationError as exc:
        errors += [
            line.strip(" -")
            for line in str(exc).splitlines()
            if line.strip().startswith("-")
        ]

    # 2. the fields a catalog needs beyond the specification, and the checksums
    errors += [f"missing {k}" for k in REQUIRED if not rec.get(k)]
    if rec.get("license") and not str(rec["license"]).startswith("http"):
        warnings.append(
            "license is not a URL; a URL is what tools and catalogs resolve"
        )
    files = rec.get("distribution", [])
    for d in files:
        if (
            d.get("@type") == "cr:FileObject"
            and d.get("sha256")
            and d.get("contentUrl")
        ):
            local = files_dir / d["contentUrl"]
            if not local.exists():
                warnings.append(
                    f"file {d.get('@id', '?')}: {d['contentUrl']} not found beside the record; checksum not verified"
                )
            elif sha256(local) != d["sha256"]:
                errors.append(
                    f"file {d.get('@id', '?')}: sha256 does not match {local.name}"
                )

    sets = rec.get("recordSet", [])
    n_fields = sum(len(rs.get("field", [])) for rs in sets)
    print(
        f"{path.name}: {rec.get('name', '?')} version {rec.get('version', '?')}; {len(files)} file(s), {len(sets)} record set(s), {n_fields} field(s)"
    )
    for w in warnings:
        print(f"  warning  {w}")
    for e in errors:
        print(f"  error    {e}")
    print(f"{len(errors)} error(s), {len(warnings)} warning(s)")
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(main())
python croissant_check.py lfs_occupation_ml.croissant.json

Output for the running example:

lfs_occupation_ml.croissant.json: LFS occupation coding (example) version 1.0.0; 2 file(s), 3 record set(s), 15 field(s)
0 error(s), 0 warning(s)

The same check on a folder in which the splits file was edited after the record was written:

lfs_occupation_ml.croissant.json: LFS occupation coding (example) version 1.0.0; 2 file(s), 3 record set(s), 15 field(s)
  error    file splits-file: sha256 does not match splits.csv
1 error(s), 0 warning(s)

  1. Run the check in the release folder. An error blocks the release.

  2. Validate the record with the reference implementation before the first publication and after any change to the generator. It checks the record against the full specification, which the script does not:

    pip install mlcroissant
    mlcroissant validate --jsonld lfs_occupation_ml.croissant.json

    For the running example the validator reports no error.

  3. Load the dataset from the record and read a few records, to confirm that the fields and the splits come through:

    import mlcroissant as mlc

    ds = mlc.Dataset(jsonld="lfs_occupation_ml.croissant.json")
    records = list(ds.records(record_set="records"))       # 600 records
    splits = list(ds.records(record_set="record_splits"))  # 600 rows: record, split

Worked example. The second output above is what happened when the data manager corrected one split assignment by hand in the splits file after the record had been written. The check reports that the digest of the splits file no longer matches, and the release is blocked. The manager regenerates the record with the build script, which recomputes the digest, and the check passes. The reference validator is then run once, reports no error, and the loading test returns 600 records and 600 split rows, with the isco_code field holding the four-digit codes.

What this does not do. The check reads structure and checksums. It does not judge whether the description is accurate or the licence is right, which the card review of chapter 4 and the licence decision of chapter 7 do.

Recipe

3.3 Publish the record where loaders look

Level
AI-ready
Skills
Catalog administrator; developer
Indicative time
Half a day

Result: the Croissant record is found by the tools and the people that look for it, on the dataset's page, in the catalog's API, and in the dataset repositories the organization uses.

  1. Put the record in the release folder beside the files, and link it from the dataset page with a line that says what it is.
  2. Embed the record in the dataset page as a script element of type application/ld+json. Because a Croissant record is a schema.org Dataset record, this is the same markup the dissemination cookbook puts on series pages, with the loading information added.
  3. Where the organization publishes the dataset in a public repository such as Hugging Face, upload the files and the card; the repository generates its own Croissant record from them, and the organization's record on the catalog page stays the authority.
  4. Register the record's URL in the catalog record (the Metadata Editor exports Croissant from a documented dataset, and the field for the link exists in the schema).

Worked example. The dataset page at the catalog embeds the record as a script element of type application/ld+json, so that a dataset search engine reads the same name, description, licence, and files as a loader does. The page also offers the record as a file with the sentence "Loading record (Croissant): tools such as mlcroissant read this file to fetch and parse the dataset". The catalog record's link field points to it. On the public repository, the uploaded files and card produce the repository's own Croissant record, and the organization's page states that the catalog's record is the authority.

What this does not do. Publication makes the record findable. Whether loaders use it is seen when a partner loads the dataset without asking a question, which the organization learns from support requests and from the use tracking of chapter 8.

Implementation options​

  • Generating the record: the script of this chapter from a dictionary; the Metadata Editor's export; a repository's automatic generation from uploaded files; mlcroissant's Python API for datasets with several files.
  • Validation: the structural check of this chapter on every release; mlcroissant's validator before publication.
  • Where it lives: the release folder, the dataset page as JSON-LD, the catalog record's link, the repository.

World Bank examples​

  • The Metadata Editor exports Croissant, schema.org, and DCAT from one documented record, which keeps the loading record and the catalog record in step.
  • The program's standards and metadata work lists Croissant with DDI, SDMX, DCAT, and schema.org as the formats an AI-ready catalog emits.

Resources​

Common mistakes​

  • A record written by hand and never regenerated. Its checksums stop matching the files at the first correction.
  • A licence as a name. A URL is what tools resolve; "CC BY 4.0" as text is a warning in the check.
  • Splits described only in the card. A loader that reads the record cannot find them; the split record sets are what it reads.

Verification​

  • Record. The record is generated from the dictionary, present in the release folder, and its checksums match.
  • Validation. The record passes the structural check on every release and the reference validator before publication.
  • Loading. The dataset loads from the record with the reference library, and the label field holds the codes.

Checklist​

0 of 5 done

Ticks are saved in this browser only.