0.3.10

This release is aimed at controllable entropy and support for running Jepsen inside Antithesis: a deterministic simulation testing environment. A new supporting library, jepsen.generator, provides the current generator system along with jepsen.random: a new namespace for pluggable random value generation. Jepsen uses these RNGs throughout, which makes it possible to run a test with a deterministic seed, or to source entropy from an external system, like Antithesis. The jepsen.antithesis library provides additional support for assertions, randomness, and lifecycle operations, plus wrappers for clients and checkers.

Also, this release introduces a new kind of visualization: op color plots, which show operations over time with different user-defined colors. This is particularly helpful for getting a feeling for "when did we lose data?" or "did only read-only queries succeed during a partition?"

New Features

• nemesis.file now supports a new :f :truncate-file operation`
• generator is now its own library
• random: a new namespace for controllable, optionally deterministic random values. All Jepsen random calls now go through this library.
• jepsen.antithesis: a new supporting library for running tests inside the Antithesis testing environment
• checker/op-color-plot: draw plots of operations over time, with different kinds of operations in different colors

Minor Improvements

• control.util/ls now supports :recursive? and :types options, which makes it easier to (e.g.) pick a random data file for a database
• nemesis.membership: document an example of how to provide :perf and :final-generator in packages
• control.net/ip: now uses the control node's DNS, then falls back to SSH. This makes ip usable when we have no SSH access.
• checker: all reify instances are now defrecords. This makes them print nicer, and allows doing some interesting tree-walking and rewriting of checkers.

Dependency Updates

• Clojure 1.12.3
• tools.cli 1.2.245
• elle 0.2.5
• jepsen.generator 0.1.0
• knossos 0.3.13
• http-kit 2.8.1
• ring 1.15.3
• sshj 0.40.0
• hiccup 2.0.0
• data.codec 0.2.0
• LazyFS 0.3.0

0.3.9

A medium-sized release, this version makes several quality-of-life improvements. We (finally!) download log files only once, rather than twice at the end of a run; this should make multi-GB log downloads less painful. A new namespace, db.watchdog, supports restarting flaky DBs automatically. Generators are a little more sophisticated, and a little easier to compose. Several error messages are nicer. Enjoy!

Bugfixes

• Jepsen now downloads log files only once, rather than twice at the end of a run
• nemesis.file: converts rational :probability values to floats, rather than letting corrupt-file silently mis-parse the rational.

API Changes

New Features

• db.watchdog: automatically restarts DB nodes when they crash
• cli/without-defaults-for: strips defaults from option specifications. This significantly streamlines using the same option spec for both test and test-all.
• control.util/tarball!: creates a tarball of a directory. Particularly helpful for downloading data dirs as a part of db/log-files.
• generator/map: functions can now take (f op test context) for more control.
• generator/shortest-any: like any, but ends as soon as any single generator does. Helpful when you have a fixed-length generator, and you want to mix in some other operations, but don't know for how long.

Minor Changes

• lein test now runs integration tests by default. I'm not entirely sure about this one; will try it out and see.
• When downloading log files, Jepsen tries to kill all DB nodes first. This reduces the chances of files shifting as we read them.
• generator: many constructors now pass through nil generators intelligently, returning nil immediately. This makes it easier to determine whether a generator will do anything before the test starts.
• nemesis/node-start-stopper: re-use existing SSH connections rather than opening new ones
• reconnect/with-conn: throws if no connection is available. This is a significantly clearer error message.
• Improved error messages for ex-infos thrown in generators and from a test as a whole
• nemesis.file: returns [:io-error ...] instead of throwing on IO errors. This cuts down on noise when trying to corrupt files the DB is (e.g.) deleting out from under us.
• nemesis.combined: shorter printed representations for some generators
• Jepsen's interrupt handler logging has more informative log lines
• reconnect: fixed a docstring typo

Dependencies

• elle 0.2.4
• jepsen.history 0.1.5
• knossos 0.3.12
• ring 1.14.1
• test.chuck 0.2.15

0.3.8

The centerpiece of 0.3.8 is a new nemesis for corrupting files: jepsen.nemesis.file. This nemesis can be scoped to specific regions of a file, is aware of chunk structure (e.g. database pages) and can be used to snapshot, restore, shuffle, and introduce bitflips in chunks stochastically. Its faults can also be scoped to specific nodes, or striped helically around a cluster. A big thank you to Ellen Marie Dash (@duckinator), who reviewed and contributed C expertise on this project.

You'll also find a handful of new utility functions, including a fast Zipfian PRNG, and a variety of bugfixes and improvements for clock skew nemeses.

Bugfixes

• SystemD took over clock management from NTP on newer Debian installs, and has its own way to disable clock sync. We now disable both NTP and systemd's time daemon.
• nemesis/compose offers better error messages when given an unknown :f

API Changes

• nemesis.time's bump operations used to generate lots of very small clock (e.g. 10 ms) adjustments, and only a few large ones. This made it hard to find errors with large (e.g. 10+ second) clock skew. We now have a higher chance of picking a large offset.

New Features

• nemesis.file: a new tool for injecting file corruption.
• generator/concurrency-limit, which limits the number of threads executing ops from a given generator to at most n.
• jepsen.web will try to shell out to the system's zip command when building zip files. This can be noticeably faster than the built-in Java zip.

Minor Changes

• nemesis.time's C-compiling powers now live in nemesis, where they're used by nemesis.file.
• util/minority: like majority, computes just shy of half an integer
• util/zipf: generates Zipfian-distributed integers
• util/rand-distribution can generate Zipfians as well
• os.debian now includes build-essential, rather than installing GCC on-demand
• checker.perf can now take configurable filenames for latency graphs, in case you want to spit out custom latency plots
• Rate graphs now skip over empty values of [:f :type]
• Knossos 0.3.11
• Ring 1.13.0
• Fipp 0.6.27
• Small documentation fixups

v0.3.7

A very small bugfix release. Jepsen.history incorrectly threw IndexOutOfBoundsException when asked for an out-of-bounds index with a default value. Clojure's semantics are to return the default value, which we now match. This fixes specific kinds of destructuring bind on empty histories.

v0.3.6

This is a sizeable release. It includes a significant correctness bugfix for a rare condition that could make operations in the history print with the wrong data. It also adds a new namespace for composing databases, nemeses, and generators when working with systems where each node has a different role. Kafka-style tests gain new powers and are significantly faster. And we have the usual slew of small bugfixes, dependency bumps, and quality-of-life improvements. Happy testing!

Bugfixes

• generator/fill-in-map no longer generates Ops with duplicate fields in their record extmaps. This fixes a rare bug where operations which used extra fields could wind up with two different values for (e.g.) (:value op) vs (pprint op). It should also improve speed and size on disk.
• checker.perf/with-range: fix a bug causing plots with zero data points to convert the plot to a string. This was expensive if the plot is large, and caused very confusing error messages. We now provide a short string message instead.
• net/iptables now handles the new error message from tc qdisc when calling net/fast!

API Changes

• control/exec will now throw a :nonzero-exit error when an exit status code is nil. Yes, this is apparently a thing that's possible.
• generator.test/with-fixed-rand-nth has been replaced by with-fixed-rands, which controls rand, rand-int, and rand-nth.
• tests/kafka: failed and info operations are now assumed to roll back consumer positions, rather than advancing them.
• tests/kafka: emit subscribe/assign ops only 1:64 ops, rather than 1:8. Now tunable via (:sub-p test).

New Features

• A new namespace, jepsen.role, supports systems where different nodes run different software.
• db/map-test wraps a DB in another which alters the test map. Helpful for composing DBs together which expect different things from their test maps.
• generator/each-process: like each-thread, this facets an underlying generator into a distinct one for each process.
• tests/kafka checks for transactions which read their own writes prior to commit.

Minor Changes

• os/centos now uses dpkg 1.19.8
• control.net/ip* now prefers v4 addresses
• control/on-nodes no longer spawns a future when given a single node--slightly more efficient.
• SSHJ now falls back to other auth methods after an AgentProxyException
• generator/map and f-map now return nil when given a nil generator, which simplifies some before-run checks.
• generator.test/default-test now includes a pair of :nodes, for generators that use nodes
• checker/check-safe now writes exceptions as data to the :error field of the results, rather than an unreadable string stacktrace
• tests.kafka now detects all duplicates even when given inconsistent offsets. It's nice to have both, it turns out.
• tests/kafka includes an :unseen key in poll operations to help operators track how far behind we are
• tests/kafka: new tests for the checker & generator
• tests/kafka: duplicate errors now include specific offsets
• tests/kafka: inconsistent-offsets errors now emit sorted sets, for readability
• tests/kafka is roughly 8x faster now, thanks to a slew of performance improvements
• tests/kafka also ignores the new cycle-exists variants of G0, G1c, etc.
• Jepsen's internal tests log less noise now
• Clojure 1.12.0
• tools.logging 1.3.0
• tools.cli 1.1.230
• unilog 0.7.32
• elle 0.2.2
• http-kit 2.8.0
• ring 1.12.2
• sshj 0.39.0
• data.codec 0.2.0
• data.fressian 1.1.0

Full Changelog: v0.3.5...v0.3.6

0.3.5

This is a relatively small release. It incorporates a new version of Elle which brings dramatic performance improvements, and has a few quality-of-life improvements.

Bugfixes

• os.debian/install and remove now acquire locks, which means you can do multiple debian-package-affecting operations concurrently against a single node.
• Thread/sleep callsites now have explicit integer coercion, fixing a crashes in newer JVMs/Clojure.

API Changes

• control.util/grepkill! now matches the full pattern of the process, rather than just the first 15 characters. This is particularly helpful for killing, say, one out of several java processes.
• net/Net has been moved to net.proto/Net. All its functions are still available in jepsen.net too.

New Features

• checker.perf: nemesis specifications can now include :hidden? true, which prevents them from appearing on graphs.

Minor Changes

• fs-cache/deploy-remote! returns the remote path it uploaded, making it easier to thread into other expressions.
• control.util/install-archive! now has docs that explain the use of file:// URLs.
• net/drop! now uses existing SSH connections, making it faster
• net/drop! has a clearer docstring
• Elle 0.2.1
• SSHJ 0.38.0
• Ring 1.11.0

0.3.4

This is a small bugfix & performance release. Just a little faster, a little more correct, a little easier to use. :-)

Bugfixes

• control.util/await-tcp-port no longer logs a truncated error message
• tests.kafka no longer crashes when checking histories without any received messages
• jepsen.independent's generators properly unlift the :value fields of the operations they pass through to underlying generators

Removals

• os.debian/install-jdk8! is gone now. The repos it relies on haven't worked in years.

Minor changes

• independent/checker uses a concurrent fold for breaking apart histories in fewer passes. This roughly doubles throughput in tests with lots of independent keys.
• independent/checker now returns results in a sorted map, which is easier to read
• generator.interpreter-test now tests matching open/close! invocations
• lazyfs version 0.2.0
• store.format logs more informative errors when serialization fails. You'll get a path to the specific element that couldn't be serialized, as well as its class. This makes serializing tests with new datatypes much less frustrating.

v0.3.3

This release updates Jepsen to run with Debian Bookworm. It also includes performance improvements aimed at testing large histories. Jepsen can run and check histories of up to a billion operations now.

Significant API Changes

• During test setup, (:generator test) is now wrapped in a new Forgettable reference type. You can deref this if you want access to the generator for some reason, but be aware that retaining the head of the generator often causes linear memory consumption during the test.
• After the test starts generating operations, attempting to deref (:generator test) will throw.
• core/run-case! and generator.interpreter/run! now return tests, rather than just histories.

Performance Improvements

• Jepsen no longer retains the head of the generator. This dramatically improves memory consumption on long-running tests: running tests of a billion operations in a 512 MB heap is entirely reasonable.
• Elle 0.1.7 comes with significant speed and memory improvements to the G1A, G1b, and internal checkers for list-append and rw-register.
• Jepsen.history is much faster to execute folds on large (e.g. 100+ million op) histories. We converted a quadratic-time loop to linear.

Bugfixes

• control.net/ip filters out loopback interfaces. Bookworm started returning 127.x.x.x interfaces from getent ahosts on some platforms.

Minor Changes

• os.debian now refers to the new netcat package name.
• nemesis.time allows ntpdate to fail during setup/teardown (which now happens on Bookworm).
• nemesis.time no longer tries to use ntpdate -p, which is deprecated in Bookworm
• tools.cli 1.0.219
• elle 0.1.7
• jepsen.history 0.1.1
• http-kit 2.7.0

Full Changelog: v0.3.2...v0.3.3

0.3.2

This is a relatively small release with a few minor bugfixes and tweaks.

Bugfixes

• control.util/wget! correctly throws exceptions when encountering unrecoverable failures.
• generator.context Contexts now correctly handle assoc.

New Features

• client/timeout wraps an existing client in a new one that times out all operations after some time.

Minor Changes

• net/net-dev uses ip, rather than /sys/class/net, for identifying network interfaces
• Ring 1.10.0
• SSHJ 0.35.0

0.3.1

This is a very small bugfix release to follow up on 0.3.0. It fixes a bug which broke analyses of tests using jepsen.independent/checker.

Bugfixes

• independent/checker now constructs jepsen.history/History objects, rather than vectors.