Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

2. Snapshot extraction

Question: How are figures and tables extracted as images?

Rationale​

PDF pages are rendered as images in memory before the detector runs. The detector returns a label, confidence score, and bounding box with normalized coordinates for each retained region. Normalized coordinates describe the left, top, right, and bottom edges as fractions of page width and height, measured from the top-left corner.

The production workflow crops these rectangles directly and saves each region as a PNG file. It does not pad boxes, merge panels, or persist page coordinates.

Target state​

  • The snapshot-only command saves inspectable PNG files.
  • The curator can distinguish the target data snapshots from irrelevant detections.

Maturity levels​

LevelSteps
FoundationalA curator identifies analytical regions on source pages and saves or inspects example data snapshots manually.
AI-readyA developer runs the documented detector, retains its candidate PNG outputs, and checks selected data snapshots against their source pages before further processing.
AI-nativeAn organization integrates detection into document intake and checks candidate relevance and data snapshot completeness under human review. Automatic quality monitoring is not supplied by this example.
Running example

The snapshot-only run processes the twelve-page sample PDF used in the tutorial and saves seven figure-labelled candidate images and one table-labelled candidate image. The cover photograph is figure_000. The chart figure_001 and demographic table table_000, both from PDF page 4, are the two data snapshots selected for the next chapter.

Recipes​

Recipe

2.1 Run snapshot-only extraction

Level
AI-ready
Skills
Developer and data curator
Indicative time
Ten minutes to prepare the command; detector execution time varies

Result: The document has been processed and snapshot files are present in the output directory.

The following command uses the PDF acquired in chapter 1 with DocLayout-YOLO on CPU. The pipeline renders each page at 300 DPI by default, which is suitable for this purpose. All twelve pages are processed.

python -m ai4data.data_snapshot.extract_data_snapshot \
--input=data/policy_brief_on_shona_community_from_stateless_to_citizens.pdf \
--output-dir=output/cookbook-data-snapshot \
--device=cpu

If a compatible NVIDIA GPU and CUDA runtime are available, select GPU inference with --device=cuda:0. The recorded example uses CPU. An unavailable CUDA device causes an error; there is no automatic CPU fallback.

  1. Use a fresh destination so that an earlier file does not cause the entire PDF to be skipped.
  2. Run the command and inspect progress and printed paths. A directory can also be passed to the --input parameter which will search recursively for PDFs.
  3. Keep source filename stems unique within a run, including across subdirectories; outputs are flattened into one model directory.

Worked example. The recorded run saved figure_000.png through figure_006.png and table_000.png, each prefixed by the complete PDF stem. figure_000 is the cover photograph, figure_001 is the chart on PDF page 4, and table_000 is the demographic table on that same page. These counts describe saved detector outputs. The photograph demonstrates why human review is still needed.

Under the tutorial's output directory, these files have the following layout:

output/cookbook-data-snapshot/
└── doclayoutyolo/
└── snapshots/
├── policy_brief_on_shona_community_from_stateless_to_citizens_figure_000.png
├── policy_brief_on_shona_community_from_stateless_to_citizens_figure_001.png
├── policy_brief_on_shona_community_from_stateless_to_citizens_figure_002.png
├── policy_brief_on_shona_community_from_stateless_to_citizens_figure_003.png
├── policy_brief_on_shona_community_from_stateless_to_citizens_figure_004.png
├── policy_brief_on_shona_community_from_stateless_to_citizens_figure_005.png
├── policy_brief_on_shona_community_from_stateless_to_citizens_figure_006.png
└── policy_brief_on_shona_community_from_stateless_to_citizens_table_000.png

What this does not do. This command does not generate metadata or persist page coordinates, confidence scores, page images, or a quality report. No OCR stage is performed.

Recipe

2.2 Inspect the figure and table data snapshots

Level
AI-ready
Skills
Developer and data curator
Indicative time
Fifteen minutes to inspect the two selected data snapshots

Result: The two examples have been checked against their source page.

The figure counter and table counter are independent and start at zero. They continue across the document, so the printed Figure 1 can have the filename suffix figure_001. The index does not encode a page number or the printed caption.

  1. Open the source page, the figure data snapshot, and the table data snapshot.

    PDF page 4 containing the chart and demographic table

  2. Check the chart's series legend, horizontal period labels, vertical percentage scale, and both population-group panels.

    Actual chart data snapshot with two population panels and an employment-status legend

  3. Check that the table retains its five demographic rows and both year columns. Compare the image with the page to identify excluded captions or source notes.

    Actual demographic table data snapshot with 2019 and 2024 columns

  4. Record quality observations separately and preserve the produced data snapshot.

Worked example. Both selected data snapshots retain their main analytical content but exclude their printed captions and source notes. This is a limitation: the later metadata extraction stage receives only that image and cannot be expected to recover those excluded details.

What this does not do. A visual comparison of two selected data snapshots is not a document-level precision or recall evaluation.

The two selected PNG data snapshots now provide the image inputs for chapter 3. The full detector-output folder remains available for inspection; selecting these two images does not remove or reclassify any of its other candidates.

Implementation options​

  • Pass a directory to --input, such as --input=data/, to search recursively for PDFs. Source filename stems must remain unique across subdirectories.
  • Use --device=cpu for CPU inference or --device=cuda:0 when a compatible NVIDIA GPU and CUDA runtime are available.
  • Add --force to replace existing outputs for the input PDFs. It deletes matching data snapshots (and metadata, if any) before extraction. A failed run does not restore those files. Use a new output directory to preserve earlier results. Chapter 6 explains recovery and rerun behavior.
  • Use --dpi to pass a different rendering resolution (default: 300). Higher resolution consumes more rendering memory and storage.

Resources​

  • The production README documents installation, supported commands, and Python interfaces.
  • The schema reference explains the metadata fields and validation boundaries.
  • The benchmark paper introduces data snapshots and reports detection and spatial accuracy metrics of the included models.

Common mistakes​

  • Reading the filename index as a printed figure number points to the wrong artifact.
  • Counting every figure-labelled candidate image as analytical content includes irrelevant detections.
  • Missed context can make the data snapshot incomplete.

Verification​

  • Inspect the printed paths and the output/cookbook-data-snapshot/doclayoutyolo/snapshots/ directory. The recorded example contains seven figure-labelled PNGs and one table-labelled PNG; investigate differences before relying on the example filenames.
  • Open figure_001 and table_000 beside PDF page 4. The selected chart should retain its panels and legend, and the table should retain its five rows and both year columns.
  • Check the page beyond each data snapshot boundary. The recorded data snapshots omit their printed captions and source notes, while the cover photograph is an irrelevant analytical candidate.

Checklist​

0 of 3 done

Ticks are saved in this browser only.