Benchmarks

Every number here was measured, and each table says on what. Throughput depends heavily on the file, the filesystem, and how many banks the analysis touches — so treat these as evidence that the approach works, not as a promise about your workload. Reproduce with bench_par on your own data.

Headline: by-bank partial decompression on JLab ifarm

A 29.7 GB CLAS12 skim file (pi0_skim_CxC_Outbending_slice000.hipo, 1.85 M events, 29,430 records) on ifarm2401 (64 logical cores). bench_par reads REC::Particle.rows() only — exactly the partial-decompression case the by-bank format is designed for.

Location	Format	Sequential	par=10	par=32	par=64	Size
/volatile (Lustre, hot)	Lz4 baseline	72 kev/s	1,437	—	—	29.7 GB
/volatile (Lustre, hot)	by-bank	437	14,724	27,643	36,578	6.66 GB (−77.6%)
/scratch (local SSD)	Lz4 baseline	159	1,112	—	—	29.7 GB
/scratch (local SSD)	by-bank	1,695	7,558	—	—	6.66 GB

(rates in kev/s)

par=64 on /volatile reaches 36.6 Mev/s — 25× the Lz4 baseline throughput at par=10.

Measured on the original by-bank variant

These were measured on the first by-bank format (fast default-LZ4 streams). Lz4PerBank — what you write today — shares the layout with HC-compressed streams: selective-read speed is the same (the figures carry over) and the file is smaller than the 6.66 GB shown. Lz4PerColumn inflates at column granularity and is smaller still.

The −77.6% is not typical

That compression ratio is exceptional for skim files: near-identical per-event topology gives the per-bank streams enormous cross-event redundancy to dedup. On generic reco files expect closer to ±5%.

Reading the matrix:

• /volatile beats /scratch in parallel here because the by-bank file was ifarm-page-cache-hot from a just-completed recook. Cold-read Lustre numbers (after the cache evicts) land closer to the /scratch row.
• Sequential is permanently Lustre-bound on /volatile — single-stream RPCs cap around 400–500 kev/s regardless of format. Stage to /scratch for sequential work.
• Thread scaling is linear well past num_cpus for the by-bank format on Lustre.

Compression modes — all formats

50,000 events of a real CLAS12 file (rec_clas_022083, 274 banks) re-encoded into every format; Apple M4 Pro, single thread, warm cache, best-of-3. Ratio is file size versus None (smaller is better). The read columns give the ms to read every value of every column of that many banks, for every event — sel = 1 bank, up to all = all 274.

Rust (bench_read_compression):

Format	Size MB	Ratio	sel (1 bk)	40 bk	all (274)
None	1734	1.00×	158	931	1589
Lz4	1081	0.62×	396	1203	1817
Lz4Best	922	0.53×	395	1198	1826
Gzip	852	0.49×	2878	3717	4348
Lz4PerBank	872	0.50×	86	1032	1529
Lz4PerColumn	813	0.47×	75	839	1280

(read columns in ms)

Two things stand out: Lz4PerColumn is the smallest file (0.47×, beating even Gzip) and the fastest read at every scope — reading one bank is ~5× faster than whole-record Lz4 (75 ms vs 396 ms) because it inflates only that bank's columns, not the record. Gzip packs tightly but is an order of magnitude slower to inflate.

Python (bench_compression.py, reading the same files through the binding — sel = arrays("REC::Particle"), all = arrays(filter_name="*"), 20k-event window):

Format	Size MB	Ratio	sel (1 bk)	all (274)
None	1734	1.00×	23	250
Lz4	1081	0.62×	48	271
Lz4Best	922	0.53×	36	260
Gzip	852	0.49×	126	343
Lz4PerBank	872	0.50×	12	171
Lz4PerColumn	813	0.47×	12	172

(read columns in ms)

The partial-decompression win reaches Python too: arrays("REC::Particle") is ~4× faster on the per-bank / per-column formats (12 ms) than on whole-record Lz4 (48 ms).

Python vs Rust

Reading REC::Particle px,py,pz,pid from a 9.1 GB CLAS12 file (598,738 events, 4.7 M particles; Apple M4 Pro, all cores, warm cache):

	Throughput	vs Rust
Rust read_columns	6.3 GB/s	1.00×
Python read_columns (NumPy)	5.8 GB/s	0.91×
Python arrays (Awkward)	5.6 GB/s	0.89×

The per-event decode runs in Rust behind a released GIL and columns move into NumPy zero-copy, so the binding costs about 10%. Full method and reproduction: Python vs Rust benchmark.

Multi-process reading

workers=N targets the per-process I/O ceiling on a parallel filesystem. On a page-cached local file it is slower — the bottleneck there is decode, not I/O, so extra processes only add spawn and IPC cost. Measured on the same 9.1 GB file (Apple M4 Pro, page-cached), workers=1 at 1.41 s beats workers=2 at 1.71 s.

This is the design working as intended, not a regression: see Parallel reading. Benchmark workers= on the farm, not on your laptop.

Baseline sequential scan

Validated on a 1.7 GB CLAS12 file (rec_clas_022050.evio.00000.hipo): a sequential Chain::events() scan reads all 187,941 events at ~257 kev/s. Chain::for_each fans the same scan across cores.

Event-tag filtering

Filtering by the per-event tag reads a single word from the event header (or the record directory) — no bank is inflated — so it can only save work. The cost in the other direction is nil: comparing an unfiltered for_each against an all-pass event_tag_any filter that runs the tag check on every event (M4 Pro, single thread, warm cache) shows the check is below measurement noise — Δ ≈ 0 ns/ev on both the Lz4 (~7.9 ns/ev) and Lz4PerColumn (~4.2 ns/ev) paths. Reads that don't use tags never enter the filter, so they are unchanged. Full table and method: Event tagging.

Reproducing

# Rust throughput, threads = 0 → all cores
cargo run --release --example bench_par -- /path/to/file.hipo 0

# Event-tag filter overhead (self-contained: writes a synthetic tagged file)
cargo run --release --example bench_event_tags -- 200000 15

# Convert a file first if you want the by-bank (Lz4PerBank) numbers
cargo run --release --example recook_by_bank -- in.hipo out_by_bank.hipo
cargo run --release --example bench_par -- out_by_bank.hipo 0

# Python vs Rust columnar read
cargo run --release --example bench_columns -- /path/to/file.hipo
python py/examples/bench_columns.py /path/to/file.hipo

# All compression modes: size + read speed at growing scope.
# Rust — the first 50k events, best-of-3; keep the per-format files so the
# Python side can read the exact same data:
OXIHIPO_BENCH_KEEP=/tmp/fmt \
  cargo run --release --example bench_read_compression -- /path/to/file.hipo 3 50000
# Python — read those same files through the binding (20k-event window):
python py/examples/bench_compression.py /tmp/fmt 20000 3