Module stream_progress

Module stream_progress 

Source
Expand description

The per-batch progress checkpoints a live parquet-stream parse reported, cached so a later cache hit can replay the SAME numbers.

progress events are in units of geometry JOBS, not meshes: the pipeline calls its batch callback with processed_jobs / total_jobs (processing/src/processor/mod.rs), and a job can produce several meshes or none. A replay that counted the meshes it emitted would therefore report different units AND a different total from a miss on the same file, over the same public event. Nothing in the geometry blob records the job counts, so the live path writes them here and the replay reads them back.

Structsยง

StreamProgress ๐Ÿ”’
What a live parse reported, in job units.
StreamProgressRecorder ๐Ÿ”’
Collects StreamProgress from the live event stream.

Functionsยง

cache_stream_progress ๐Ÿ”’
Store the checkpoints beside the geometry. Best-effort: a failure here costs the next cache hit its job-unit progress, not its correctness.
load_stream_progress ๐Ÿ”’
Read back the checkpoints for a cache hit. None for an entry written before this sidecar existed, or one that no longer deserializes.
stream_progress_cache_key ๐Ÿ”’
Cache key for the checkpoint sidecar, keyed alongside the geometry and metadata entries for the same request.