Skip to content

The ReplicaOfflineMsg grace period ends at the first forward, so a second replication server can be left unaware of the offline replica #917

Description

@vharseko

DSRSShutdownSync tracks one pending ReplicaOfflineMsg per replica, and the first
ServerWriter to publish it clears that entry for everybody.

The path

ReplicationServerDomain.put()
(opendj-server-legacy/src/main/java/org/opends/server/replication/server/ReplicationServerDomain.java:361-373)
pushes a message received from a DS to every connected replication server:

if (sourceHandler.isDataServer())
{
  for (ReplicationServerHandler rsHandler : connectedRSs.values())
  {
    if (!isDifferentGenerationId(rsHandler, updateMsg))
    {
      addUpdate(rsHandler, updateMsg, notAssuredUpdateMsg, assuredServers);
    }
  }
}

Each of those handlers has its own ServerWriter, and whichever publishes first calls
replicaOfflineMsgForwarded()
(opendj-server-legacy/src/main/java/org/opends/server/replication/server/ServerWriter.java:113-117),
which removes the single entry the wait of
ReplicationServerDomain.shutdown() is watching.

What it costs

With RS2 and RS3 both connected, the message is enqueued for both. RS2's writer publishes it,
the entry disappears, awaitReplicaOfflineMsgForwarded() returns, and stopAllServers(true)
runs while RS3's writer has not drained its queue yet - MessageHandler.shutdown() then does
msgQueue.clear() and the session is closed, so RS3 never learns the replica went offline. Its
ChangeNumberIndexer keeps the medium consistency point pinned to the last CSN of that replica
(ChangeNumberIndexer.replicaOffline() / getOldestLastAliveCSN(),
opendj-server-legacy/src/main/java/org/opends/server/replication/server/changelog/file/ChangeNumberIndexer.java:197-212),
so its change number index and external changelog stop advancing for that domain until the
replica comes back.

Not obvious to fix

The granularity is inherited from the original OPENDJ-1453 design, where the flag was a plain
set of base DNs and the only reader was a spin in ServerWriter; it became load bearing when
the wait moved to ReplicationServerDomain.shutdown() in #900. Tracking it per handler means
deciding what to do with a peer that disconnects, or joins, during the grace period.

Found while fixing #900, which leaves this granularity as it is.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions