Zero-copy columnar reads
bank.col::<T>("name") hands back a Cow<[T]> borrowed straight from the decompressed record buffer when the bytes are aligned, with a one-shot copy fallback otherwise. Fixed-length array columns (name/T#N) read as [T; N].
Bounded memory, any file size
Records stream one at a time through a recycled buffer — the file is never mapped or read whole. A sequential scan of a 100 GB file holds about one record resident; parallel scans hold one per worker.
One reader: Chain
Chain::open takes a file, a directory, a glob, or a list of paths. Multi-file chains share a single parsed dictionary, and events() yields Result, so a truncated record surfaces as an Err instead of a panic.
Data-parallel scans
for_each(threads, f) fans work across cores out of order — 0 for every core, 1 for sequential, n for exactly n. The thread count is the only difference between the two.
Decompress only what you read
Lz4ByBank stores each bank as its own LZ4 stream and inflates one only when ev.bank(name) asks. Real analyses touch a handful of ~30 banks — the rest stay compressed. No reader-side API change.
Python that feels like uproot
A HIPO bank reads like an Awkward jagged branch. The per-event loop runs in Rust with the GIL released and columns move into NumPy zero-copy, so the binding costs about 10%.
Rust
use oxihipo::{Chain, Filter};
let chain = Chain::open("/data/run5042")? // file | dir | glob | list
.with_filter(Filter::require(["REC::Particle"]))?;
for ev in chain.events() {
let p = oxihipo::or_continue!(ev?.bank("REC::Particle"));
for r in 0..p.rows() {
let pid: i32 = p.get("pid", r);
let px: f32 = p.get("px", r);
}
}
Python
import oxihipo as ox
f = ox.open("/data/run5042") # file | dir | glob | list
p = f.arrays("REC::Particle", ["pid", "px"]) # ak.Array: N * var * {pid, px}
# bigger than RAM? stream it in bounded chunks
for chunk in f.iterate("REC::Particle", step_size="200 MB"):
hist.fill(ak.flatten(chunk.px))
Measured, not claimed
A 29.7 GB CLAS12 skim on JLab ifarm (/volatile, Lustre, 64 cores), read with Lz4ByBank — the format extension that inflates only the banks an analysis actually touches.
par=64, versus 1.4 Mev/s for the Lz4 baseline