Bitruvius Imagery & 3D Suite / TurboZstd Drop-in for libzstd Shipping · Private Beta

Same zstd files. Faster to read at every level.

Zstandard is the compression inside GeoTIFFs, Zarr stores, databases and backups. TurboZstd reads and writes the same files as the reference libzstd, byte-for-byte standard RFC 8878 zstd, in memory-safe code with no C toolchain to build.

It exports the same ZSTD_* functions GDAL, libtiff, rasterio, Zarr, Python zstandard and Java zstd-jni call. Replace the system libzstd library; no recompile.

1.73–2.07×

faster to read, at every level

On Intel. 1.18×–1.49× on Apple silicon. Data is read far more often than it is written.

3.65×

faster archival writes, one core

At level 22 (1.79× at level 19), for files 2.6–2.7% larger.

19.4×

faster archival writes, every core

Against libzstd's own multi-threaded encoder at level 22, eight threads each. 17.6× on Apple silicon.

failures reading our files with libzstd

All 804 samples round-trip through the reference libzstd, on both chips, one thread and many.

The trade, up front: on one thread we write slower than libzstd at levels 1 and 3, and file size depends on your data type. Every figure is below.

The 60-second version

Zstandard is everywhere data is stored, and it is read far more than it is written.

The format

Zstandard

General-purpose compression with an open specification (RFC 8878). It sits inside imagery files, array stores, databases, logs and backups.

The incumbent

libzstd

The C reference library, linked by nearly everything that speaks zstd. It is the bar any replacement has to clear, on the same data.

The bill

Reads add up

Data is compressed once and decompressed every time it is used, so read speed is the number most systems feel, and archival writes are the big batch jobs.

The fix

A memory-safe drop-in

TurboZstd writes standard zstd that libzstd reads, reads everything libzstd writes, and replaces the library without a recompile.

What it means for the bill

Faster reads. Archives in minutes.

The Intel figures, translated.

42%

less time reading compressed data

At the slowest of the measured levels; more at the rest.

2.7 hours

for a 10-hour archival job, one core

At level 22, for files 2.6–2.7% larger.

31 minutes

for the same job on every core

Against libzstd's own parallel encoder, eight threads each.

Illustrative, from the measured Intel figures. A pilot measures yours.

The trade, stated

Where we win, and where libzstd still does.

We win

- Reading, at every level, on both chips.
- Writing from level 5 up on one thread (1.54× at level 5, 2.01× at level 9), and from level 3 up on every core on Intel.
- Size at the everyday levels: 5.2% smaller at level 1 and 2.4% at level 9; terrain models 16.6% and radar 11.3% smaller at level 1.

libzstd wins

- Writing at levels 1 and 3 on one thread, and level 1 on every core (0.99× Intel, 0.57× Apple silicon).
- Size at the archival levels, by 2.6–2.7%, where its deeper search still pays.
- Size on two data types at every level: optical imagery by up to 13.7% and AI inference masks by up to 8.9%.
- Reading two data types at level 1 on Apple silicon (hyperspectral 0.92×, radar quicklooks 0.91×).

One thread, every level

Reads faster at every level. Archival writes 1.8–3.6× faster.

One thread each against libzstd across the 804-sample corpus. Writing is slower than libzstd at levels 1 and 3, which already take microseconds, then pulls ahead at level 5 and wins 1.8× at level 19 and 3.6× at level 22. Reading wins at every level on both chips: 1.73× to 2.07× on Intel, 1.18× to 1.49× on Apple silicon.

Speed vs libzstd by compression level

Encode · AVX2 Encode · NEON Decode · AVX2 Decode · NEON — — 1× parity with libzstd

Log scale so the full range fits, from under 1× at level 1-3 writes to several-fold at level 22. Dashed = Apple silicon (NEON); solid = Intel (AVX2).

Every core

Faster than libzstd's own parallel encoder from level 3 on Intel and level 5 on Apple silicon.

Intel reaches 19.4× at level 22 and Apple silicon 17.6×. At level 1 both still trail libzstd's own multi-threaded encoder (0.99× and 0.57×). Level 9 on every core runs 3.1× faster on Apple silicon, at 2% smaller output. libzstd barely speeds up on archival work; at level 22 most raster types run 10× to 16× faster on Apple silicon, and video radar reaches 26×.

All-cores encode throughput vs libzstd (8 threads each)

Encode · AVX2 (par) Encode · NEON (par) — — 1× parity with libzstd

encode_all_par against libzstd's own multi-threaded encoder, eight threads each, across the full 804-sample corpus. Both chips at 0c15e125 (v1.1.0). No interop failures. Log scale; dashed = Apple silicon, solid = Intel. Parallel is encode-only.

Fast levels

The speed-first levels read up to 2.5× faster.

TurboZstd supports libzstd's full negative range (-1 … -131072), a speed-first tier that gives up a little compression. Reading wins on both chips and writing clears parity on both: Intel reads 2.11× to 2.49× faster and Apple silicon 1.34× to 1.68×. Results vary widely by data type, from 0.73× on radar at level -3 to 4.42× on radar quicklooks at -5. Files stay within 1.2% of libzstd overall, but optical imagery runs 22.8% larger at -5.

Negative-level throughput vs libzstd (single-thread)

Decode · AVX2 Decode · NEON — — 1× parity with libzstd

Single-thread decode, the full 804-sample corpus, weighted by bytes, measured at 0c15e125 (v1.1.0) on both chips. Levels -7 and -19 are not charted; neither board measures them. No interop failures on either.

Your data type

Your data type. Your win column.

Size and speed against libzstd for each data type we benchmark, Intel (Arrow Lake) and Apple silicon (M3 Max) side by side. The all-cores bars run our parallel encoder against libzstd's own at the same thread count. Above 1× = TurboZstd faster. Positive size = TurboZstd smaller. At level 22 on every core, data types run 4.5× to 26× faster; on one thread the best are AI masks 9.27×, video radar 5.76× and terrain 2.82×.

AI Inference

Model inference output tensors (asphalt, vegetation, etc.)

228.4 MiB corpus