Skip to content

Replication: an Error in a replay kills a replay thread the pool never replaces #923

Description

@vharseko

ReplayThread.run() wraps the take-and-replay loop in catch (Exception e) with the comment "catch all exceptions happening so that the thread never dies even in case of problems" (opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/ReplayThread.java:116-124).

An Error is not an Exception, so it does exactly what the comment says must not happen: it unwinds run() and the thread ends. MultimasterReplication.createReplayThreads() runs at initialization and on a ds-cfg-num-update-replay-threads change only, so nothing replaces it - the pool keeps a terminated Thread in its list and the shared updateToReplayQueue has one consumer fewer for every domain of the server. DirectoryThread's handler logs ERR_UNCAUGHT_THREAD_EXCEPTION and raises an ALERT_TYPE_UNCAUGHT_EXCEPTION, which is the only trace of it.

Repeated, this stops replication for the whole server: processUpdate() blocks in its offer loop once no thread takes from the queue.

Widening the catch to Throwable is not enough on its own:

Both halves want doing together, or an Error which repeats turns into a change nobody can replay while the thread which reports it lives on.

Found while reviewing #892.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions