Writing

Writing is closure-driven: a Writer builder, then one closure per event, one per bank, one per row.

use oxihipo::{Compression, Writer};

let mut w = Writer::create("out.hipo")
    .schemas(dict)
    .compression(Compression::Lz4)
    .build()?;

w.event(|ev| {
    ev.bank("REC::Particle", |b| {
        b.row(|r| {
            r.set("pid", 11_i32)?;
            r.set("px", 0.5_f32)
        })?;
        Ok(())
    })?;
    Ok(())
})?;

w.finish()?;

finish() writes the trailer and index — a Writer dropped without it leaves a file no reader will accept, so don't skip it.

The builder

Writer::create(path) returns a builder:

Method	Purpose
`.schemas(dict)`	the `Dict` describing every bank you'll write (required)
`.compression(c)`	see below (defaults to `Lz4`)
`.tag_names(names)`	persist a name↔bit tag registry (see Tagging events)
`.max_record_events(n)`	flush a record after `n` events
`.max_record_bytes(n)`	flush a record once it reaches `n` bytes
`.build()`	produce the `Writer`

The two max_record_* knobs control record granularity. Bigger records compress better; smaller records give parallel readers finer-grained units and lower the reader's resident memory.

Choosing a compression

Compression is a (codec × layout) pair — what squeezes the bytes, and what gets squeezed separately:

use oxihipo::{Codec, Compression, Layout};

// Codec:  None | Lz4 | Lz4Hc | Gzip | Zstd
// Layout: PerChunk | PerBank | PerColumn
Compression::new(Codec::Zstd, Layout::PerColumn).with_zstd_level(3)

All 15 pairs work. The six that predate the matrix keep their old names as shorthand, and mean exactly what they always did:

use oxihipo::{Codec, Compression, Layout};

Compression::None            // None  × PerChunk
Compression::Lz4             // Lz4   × PerChunk   — stock, hipo4-compatible
Compression::Lz4Best         // Lz4Hc × PerChunk   — needs the `lz4-c` feature
Compression::Gzip            // Gzip  × PerChunk   — stock, hipo4-compatible
Compression::Lz4PerBank      // Lz4Hc × PerBank
Compression::Lz4PerColumn    // Lz4Hc × PerColumn  — what `skim` defaults to
assert_eq!(Compression::Lz4PerColumn, Compression::new(Codec::Lz4Hc, Layout::PerColumn));

Pick the layout first — it decides selective-read speed, and is worth an order of magnitude where the codec is worth tens of percent on size. PerChunk must inflate a whole record to reach any bank in it; PerBank and PerColumn inflate only what you touch.

Then pick the codec on write cost. Zstd × PerColumn is the best default: smaller than Lz4Hc × PerColumn and 10.7× faster to write. Use Lz4 if writes dominate, Gzip if bytes do.

Only the four PerChunk codecs are byte-compatible with the C++ hipo4 reader; Lz4Hc × PerBank and Lz4Hc × PerColumn are readable by the feature/bybank-bycolumn-compression branches of hipo-cpp and hipo-java. The other nine pairs are oxihipo extensions those readers reject as an unknown tag — use them for Rust-only (or oxihipo-Python-only) consumers.

The measured matrix of all fifteen is in Compression formats.

Tagging events

Every event carries a 32-bit tag (EH_TAG). Set it inside the event closure with ev.with_tag(...); a reader then filters on it without inflating any bank (see Reading). A raw u32 works, but a tag_flags! block turns the bits into named physics categories:

oxihipo::tag_flags! { pub EventTag { Dvcs = 0, Sidis = 1, Elastic = 2 } }

let mut w = Writer::create("out.hipo")
    .schemas(dict)
    .tag_names(EventTag::NAMES)   // persist the name↔bit registry (optional)
    .build()?;

w.event(|ev| {
    ev.with_tag(EventTag::Dvcs | EventTag::Sidis);  // a TagSet, or a raw u32
    // ... banks ...
    Ok(())
})?;

with_tag accepts a raw u32 or a TagSet. tag_names is optional: it writes the (name, bit) table into the file's dictionary record so a consumer can resolve names without the tag_flags! declaration — chain.tag_registry() in Rust, f.tag_names / filtered(event_tag="dvcs") in Python. The registry is additive (readers that don't know it skip it) and skim copies it through, so a tagged DST stays self-describing. Writing no registry leaves the output byte-for-byte as before.

Which tag names are legal

The registry is stored as one name=bit line per entry, so a name that cannot survive that round trip is refused rather than written and silently changed. Since 0.7.0 build() fails if any name is empty, contains = or a line break, or has leading/trailing whitespace.

Before that check existed, such a name came back as something else — "a\nb" read back as "b", and " padded " as "padded" — under which every later mask() lookup quietly missed. TagRegistry::insert and TagRegistry::from_names return Result for the same reason; WriterBuilder::tag_names keeps its signature and surfaces the error from build().

Writing tagged DSTs

To produce a tagged file — classify each event and stamp the result — use Chain::skim_tagged. It copies the (filtered) chain like skim, but a classifier closure computes each event's new EH_TAG, and tag_names records the output registry so the DST rereads by name. This closes the select→label→write→reread loop:

oxihipo::tag_flags! { pub Cat { Dvcs = 0, Sidis = 1 } }

Chain::open("run.hipo")?
    .with_filter(Filter::require(["REC::Particle"]))?     // select
    .skim_tagged("dvcs.hipo", Compression::Lz4PerColumn, Cat::NAMES, |ev| {
        if is_dvcs(ev) { Cat::Dvcs } else { Cat::Sidis }  // label
    })?;                                                  // write

// reread: Chain::open("dvcs.hipo")? carries the Cat registry, so
// Filter::event_tag_any(Cat::Dvcs) — or filtered(event_tag="dvcs") in Python.

The retag touches only the event header; banks are copied through unchanged, so it is as cheap as a plain skim. The source file's own registry is not carried over (the closure defines a fresh scheme). A runnable end-to-end demo is examples/tag_and_skim.rs.

Updating a tag in place

To flip one event's tag on an existing file — without rewriting it — Chain::set_event_tag patches the 4-byte EH_TAG on disk (and set_event_tags a batch, all-or-nothing). It needs write permission on the file, and works only for uncompressed files (Compression::None): for a compressed record the tag lives inside a compressed block, so it returns HipoError::InPlaceTagUnsupported — use skim_tagged to rewrite those.

let chain = Chain::open("run.hipo")?;      // written with Compression::None
chain.set_event_tag(42, EventTag::Dvcs)?;  // one 4-byte write, no rewrite
chain.set_event_tags([(10, 1_u32), (20, 2)])?;

The event-header magic is verified before every write, so a bad index can't corrupt the file, and the change is visible to the next read immediately.

For how tags are stored on disk, a benchmark showing the pushdown is free, and what's planned next, see the event-tagging design & roadmap.

Array columns

A column can hold a fixed-length array instead of a scalar. In schema text a column type is a type letter optionally followed by #N: F#3 is three float32 per row, S#2 two int16 (F=f32, D=f64, I=i32, S=i16, B=i8, L=i64). Declare it as text, or from (name, type, length) triples where length > 1 makes the column an array:

use oxihipo::{DataType, Schema};

// text form — name/T#N
Schema::parse_text("{REC::Traj/100/1}{trk_id/I,cov/F#6,hits/S#3}")?;

// or programmatically
Schema::from_columns(
    "REC::Traj",
    100,
    1,
    [
        ("trk_id".into(), DataType::Int, 1),
        ("cov".into(), DataType::Float, 6),
        ("hits".into(), DataType::Short, 3),
    ],
);

Write a row's array with the same set you use for scalars — pass the array, and its length must match the declared N:

b.row(|r| {
    r.set("trk_id", 7_i32)?;
    r.set("cov", [0.0_f32, 0.1, 0.2, 0.3, 0.4, 0.5])?;
    r.set("hits", [1_i16, 2, 3])?;
    Ok(())
})?;

Reading them back is covered in Reading · Array columns; from Python they arrive as fixed-size sublists — see Python · Array columns. A runnable end-to-end example is examples/write_array.rs.

Fixed length only

Every row of a T#N column has the same N. Genuinely ragged per-row lengths aren't a column type — model those as separate bank rows cross-referenced by an index column (the CLAS12 pindex pattern).

C++ hipo4 cannot read T#N columns

A plain-Lz4 compatibility gap, separate from the split codecs, and not an oxihipo bug — Java parses T#N correctly and documents it as oxihipo's serialization, so C++ is the outlier.

C++'s schema::parse splits the schema text on , and / only, so for cov/F#6 the type token is the literal "F#6"; getTypeByString returns -1 and getTypeSize(-1) returns 0. Measured on hipo4 itself with pid/S,cov/F#6,px/F, that has two consequences:

the array column reads as type -1 — bank.get prints ---> error(get) : unknown type and returns 0;
every column declared after it is mis-offset, silently. px lands at byte 2 instead of 26 and the row length comes out 6 instead of 30, with no diagnostic at all.

The second is the dangerous one: it produces plausible wrong numbers rather than an error. If a C++ consumer has to read the file, keep array columns out of the schema, or split them into N scalar columns.

Copying events verbatim

append_raw(&[u8]) writes an already-encoded event through unchanged. This is what Chain::skim uses internally:

for ev in chain.events() {
    w.append_raw(ev?.bytes())?;
}

It skips decode and re-encode entirely, so a skim is bounded by I/O and recompression rather than by parsing.

The builder​

Choosing a compression​

Tagging events​

Writing tagged DSTs​

Updating a tag in place​

Array columns​

Copying events verbatim​