zstd-seekable-format-go

Package seekable adds random-access reads to compressed Zstandard data.

It writes compressed streams with a seek table and reads them by decompressed byte offset through Reader, ReadAt, and Seek. The resulting stream remains valid Zstandard data, so standard Zstandard readers can still decode it sequentially.

The wire format is the Zstandard seekable format: compressed frames followed by a seek table in a skippable frame.

The package uses small encoder/decoder interfaces and is tested with github.com/klauspost/compress/zstd.

Install

go get github.com/SaveTheRbtz/zstd-seekable-format-go/pkg

import seekable "github.com/SaveTheRbtz/zstd-seekable-format-go/pkg"

Write

func writeSeekable(dst io.Writer, chunks [][]byte) error {
	enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest))
	if err != nil {
		return err
	}
	defer enc.Close()

	w, err := seekable.NewWriter(dst, enc)
	if err != nil {
		return err
	}
	for _, chunk := range chunks {
		if _, err := w.Write(chunk); err != nil {
			return err
		}
	}
	return w.Close()
}

Writer.Close writes the final seek table. Without it, Reader and NewSeekTable cannot find the random-access metadata.

Read

func readAt(src io.ReadSeeker, off int64, p []byte) error {
	dec, err := zstd.NewReader(nil)
	if err != nil {
		return err
	}
	defer dec.Close()

	r, err := seekable.NewReader(src, dec)
	if err != nil {
		return err
	}
	defer r.Close()

	_, err = r.ReadAt(p, off)
	return err
}

Offsets are decompressed byte offsets. Reader implements io.Reader, io.ReaderAt, io.Seeker, and io.Closer.

Metadata

func frameForOffset(r *seekable.Reader, off uint64) (seekable.FrameOffsetEntry, error) {
	table, err := r.SeekTable()
	if err != nil {
		return seekable.FrameOffsetEntry{}, err
	}
	entry, ok := table.EntryByDecompressedOffset(off)
	if !ok {
		return seekable.FrameOffsetEntry{}, io.EOF
	}
	return entry, nil
}

SeekTable exposes the decompressed size, frame count, checksum flag, and frame lookup by id or decompressed offset.

Compatibility

Seekable streams are valid Zstandard streams:

func readSequential(src io.Reader) ([]byte, error) {
	zr, err := zstd.NewReader(src)
	if err != nil {
		return nil, err
	}
	defer zr.Close()

	return io.ReadAll(zr)
}