HDDS-14509. Allow client to choose the read consistency level by ivandika3 · Pull Request #9796 · apache/ozone

ivandika3 · 2026-02-20T09:51:09Z

What changes were proposed in this pull request?

Currently, if OM follower read is enabled, all OM followers will serve read requests even if the client does not enable the follower read.

For leader-only client, this might be unexpected since they expect the OM follower to throw OMNotLeaderException.

We can support some kind of OM request metadata to indicate to the OM whether the client enables the follower read or not.

The implementation introduces ReadConsistencyHint field in OMRequest which allows client to hint to the OM about what the read consistency the client wants. However, for compatibility reasons, the client ReadConsistencyHint is only supported by OM in a best-effort basis. For example, if the new client sends the hint to old OM that does not support it or OM that disables linearizable read, the OM can choose not to respect the read consistency.

Currently, client can configure the default leader and follower read consistency which applies to corresponding requests. However, the long term idea is to allow client to specify per-request consistency requirements. For example in S3 use cases, client can use custom header (e.g. "x-ozone-read-consistency") to pick the desired consistency.

The idea of client-defined consistency level is inspired by other systems and literature:

RQLite: https://rqlite.io/docs/api/read-consistency/
Consul: https://developer.hashicorp.com/consul/api-docs/features/consistency
"Replicated Data Consistency Explained
Through Baseball": https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf

Note: Since this changes require changes in protobuf definition and possibly user-facing read consistency level. Thorough reviews and suggestions are greatly appreciated. (Edit: Currently putting this to Draft first until the proto and API changes are finalized since IMO the API and proto changes are still slightly awkward).

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14509

How was this patch tested?

IT.

Clean CI run: https://github.com/ivandika3/ozone/actions/runs/22216665699

szetszwo · 2026-02-21T21:42:40Z

@ivandika3 , thanks for working on this!

This PR also includes some refactoring/renaming and the patch file is ~70KB. Could you separate the refactoring/renaming to another JIRA? It will be easier to review.

ivandika3 · 2026-02-22T02:42:52Z

Thanks @szetszwo for the review. I have removed some of refactoring by reverting rename from leaderFailoverProxy to failoverProxy. Regarding the refactoring for transferLeadershipToAnotherNode to MiniOzoneHAClusterImpl, I did it to remove duplication on this method in the new tests. Let me know if you'd prefer to revert this as well.

szetszwo

@ivandika3 , thanks for the update!

I have some high level comments inlined.

szetszwo · 2026-02-23T18:00:26Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ReadConsistency.java

+  DEFAULT(ConsistencyType.NON_LINEARIZABLE, false),
+  STALE(ConsistencyType.STALE, true),
+  LINEARIZABLE_LEADER_READ(ConsistencyType.LINEARIZABLE, false),
+  LINEARIZABLE_FOLLOWER_READ(ConsistencyType.LINEARIZABLE, true),
+  LOCAL_LEASE_FOLLOWER_READ(ConsistencyType.LOCAL_LEASE, true);


Stale read can be supported by local lease with infinite log lag and time limit. We may remove STALE for simplicity.

I think we don't need the inner enum ConsistencyType. Just add a boolean for linearizable.

isAllowFollowerRead sounds odd. Let's use nouns for the fields and add the verb to the method.

Below is my suggestion:

public enum ReadConsistency { DEFAULT(false, false), LOCAL_LEASE(false, true), LINEARIZABLE_LEADER_ONLY(true, false), LINEARIZABLE_ALLOW_FOLLOWER(true, true); private final boolean linearizable; private final boolean followerRead; ReadConsistency(boolean linearizable, boolean followerRead) { this.linearizable = linearizable; this.followerRead = followerRead; } public boolean isLinearizable() { return linearizable; } public boolean allowFollowerRead() { return followerRead; } ... }

Thanks for the review and the suggestion on streamlining the ReadConsistency. I have changed accordingly and change some namings (e.g. removed "Consistency Type" naming).

szetszwo · 2026-02-23T18:30:41Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

+          }
+          // The LocalLease lag is too high, trigger failover
+          throw createLeaderErrorException(raftServerStatus);
+        case LEADER_AND_NOT_READY:


For LOCAL_LEASE_FOLLOWER_READ, the LEADER_AND_NOT_READY case should be treated as the same as the NOT_LEADER case, i.e. just consider it as a follower. (fixed typo "LEADER_AND_READY")

Thanks for catching this, updated.

szetszwo · 2026-02-23T18:39:00Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

+          long localLeaseLagLimit = localLeaseContext.getLagLimit() > 0 ?
+              localLeaseContext.getLagLimit() : ozoneManager.getConfig().getFollowerReadLocalLeaseLagLimit();
+          long localLeaseLeaseTimeMs = localLeaseContext.getLeaseTimeMs() > 0 ?
+              localLeaseContext.getLeaseTimeMs() : ozoneManager.getConfig().getFollowerReadLocalLeaseTimeMs();


Use localLeaseContext.getLagLimit() and hasLeaseTimeMs(). We may use -1 for allowing infinite lag/time.

Thanks, used the has... and also updated allowFollowerReadLocalLease to allow infinite lag.

szetszwo

@ivandika3 , thanks for the update! Please see the comments inlined.

szetszwo · 2026-02-24T18:42:34Z

...rc/main/java/org/apache/hadoop/ozone/om/ha/HadoopRpcOMFollowerReadFailoverProxyProvider.java

+        ReadConsistencyProto defaultReadConsistency = isFollowerReadEligible
+            ? followerReadConsistencyType : leaderReadConsistencyType;
+        if (defaultReadConsistency != null &&
+            defaultReadConsistency != ReadConsistencyProto.UNKNOWN_READ_CONSISTENCY) {
+          omRequest = omRequest.toBuilder()
+              .setReadConsistencyHint(ReadConsistencyHint.newBuilder()
+                  .setReadConsistency(defaultReadConsistency).build())
+              .build();


Change them to followerReadConsistencyType and leaderReadConsistencyType to ReadConsistencyHint. So it does not need to build again and again.

if (!omRequest.hasReadConsistencyHint()) { final ReadConsistencyHint defaultReadConsistency = isFollowerReadEligible ? followerReadConsistencyType : leaderReadConsistencyType; if (defaultReadConsistency != null) { omRequest = omRequest.toBuilder() .setReadConsistencyHint(defaultReadConsistency) .build(); args[1] = omRequest; } }

Thanks for this optimization, updated.

szetszwo · 2026-02-24T19:02:12Z

hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto

+enum ReadConsistencyProto {
+  // Unknown consistency, the read consistency behavior is decided
+  // by the OM
+  UNKNOWN_READ_CONSISTENCY = 0;


It is a good idea to have this as described in https://protobuf.dev/best-practices/dos-donts/#unspecified-enum. How about calling it UNSPECIFIED?

Got it, updated. We might add some kind of suffix or prefix (e.g. UNSPECIFIED_CONSISTENCY) since AFAIK once one one enum uses UNSPECIFIED other enum within the scope cannot use UNSPECIFIED anymore. Please let me know what you think.

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ReadConsistency.java

szetszwo · 2026-02-24T19:18:27Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/Hadoop3OmTransport.java

+
  private final HadoopRpcOMFailoverProxyProvider<OzoneManagerProtocolPB> omFailoverProxyProvider;
+  private final boolean followerReadEnabled;
+  private final ReadConsistencyProto defaultLeaderReadConsistency;


Use ReadConsistencyHint

Thanks, updated.

szetszwo · 2026-02-24T19:21:21Z

...ne/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java

        .build();
  }

+  private RaftClientRequest.Type getRaftReadRequestType(OMRequest omRequest) {


Add static.

szetszwo · 2026-02-24T19:45:01Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

    }

+    if (leaseLagLimit == -1) {
+      // Allow infinite lag time, which allows unbounded stale reads


It should be "infinite log lag ~~time~~"

Thanks for catching this, updated.

szetszwo · 2026-02-24T19:46:40Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

+    if (leaseTimeMsLimit == -1) {
+      leaseTimeMsLimit = Long.MAX_VALUE;
+    }


Combine to the if-condition below:

if (leaseTimeMsLimit >= 0 && leaderInfo.getLastRpcElapsedTimeMs() > leaseTimeMsLimit) {

Thanks, updated.

The initial local lease implementation allows negative leaseTimeMsLimit which means that lease will be rejected all the time? It is tested by testAllowFollowerReadLocalLease. IMO, any negative leaseTimeMsLimit should infer infinite log and lag time? In that case, should we change the test to reflect this?

Let's make leaseTimeMsLimit >= -1 and use -1 for infinity. We should change the test.

szetszwo · 2026-02-24T20:02:57Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

  }

-  boolean allowFollowerReadLocalLease(Division ratisDivision, long leaseLogLimit, long leaseTimeMsLimit) {
+  boolean allowFollowerReadLocalLease(Division ratisDivision, long leaseLagLimit, long leaseTimeMsLimit) {


Since there are two types of lag, time lag and log lag, the original name leaseLogLimit is better.

BTW, we should change the conf

ozone.om.follower.read.local.lease.lag.limit

to

ozone.om.follower.read.local.lease.log.limit

Thanks for catching this, updated. The initial change was because of the configuration which lead me to believe it's a typo.

szetszwo · 2026-02-24T20:13:01Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

+        raftServerStatus = omRatisServer.getLeaderStatus();
+        switch (raftServerStatus) {
+        case LEADER_AND_NOT_READY:
+          throw createLeaderErrorException(raftServerStatus);


Remove this line since LEADER_AND_NOT_READY and NOT_LEADER should be the same.

Good catch. LeaderStateImpl#getReadIndex handles isReady and will wait until the leader apply the no-op entry.

szetszwo · 2026-02-24T20:14:26Z

hadoop-ozone/ozonefs-hadoop2/src/main/java/org/apache/hadoop/fs/ozone/Hadoop27RpcTransport.java


  private final HadoopRpcOMFailoverProxyProvider<OzoneManagerProtocolPB> omFailoverProxyProvider;
+  private final boolean followerReadEnabled;
+  private final ReadConsistencyProto defaultLeaderReadConsistency;


Use ReadConsistencyHint.

Thanks, updated.

…_ALLOW_FOLLOWER

szetszwo

@ivandika3 , thanks for the update! The change looks good. Just have two minor comments inlined.

szetszwo · 2026-02-25T17:33:44Z

hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto

+enum ReadConsistencyProto {
+  // Unspecified consistency, the read consistency behavior is decided
+  // by the OM
+  UNSPECIFIED = 0;


Just realized that UNSPECIFIED only can be used once. Let's use READ_CONSISTENCY_UNSPECIFIED

szetszwo · 2026-02-25T18:04:20Z

...ne/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java

+    if (!omRequest.hasReadConsistencyHint() || !omRequest.getReadConsistencyHint().hasReadConsistency() ||
+        omRequest.getReadConsistencyHint().getReadConsistency() == ReadConsistencyProto.UNSPECIFIED) {


Add a util method. Use it here and also OzoneManagerProtocolServerSideTranslatorPB.

//OmUtils public static boolean specifiedReadConsistency(OMRequest request) { return request.hasReadConsistencyHint() && request.getReadConsistencyHint().hasReadConsistency() && request.getReadConsistencyHint().getReadConsistency() != UNSPECIFIED; }

szetszwo

@ivandika3 , thanks for the update! Please see the comments inlined.

szetszwo · 2026-02-26T18:51:20Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

    }

-    if (leaseTimeMsLimit != -1 && leaderInfo.getLastRpcElapsedTimeMs() > leaseTimeMsLimit) {
+    if (leaseTimeMsLimit >= -1 && leaderInfo.getLastRpcElapsedTimeMs() > leaseTimeMsLimit) {


It should be != or > -- when setting it to -1, it should not return false here.

szetszwo · 2026-02-26T18:53:37Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OmConfig.java

@@ -178,7 +179,8 @@ public class OmConfig extends ReconfigurableConfig {
      tags = {ConfigTag.OM, ConfigTag.PERFORMANCE, ConfigTag.HA, ConfigTag.RATIS},
      description = " If the lag time Ms between leader OM and follower OM is larger " +
          "than this number, the follower OM is not up-to-date. " +
-          "By default, it's set to Ratis RPC timeout value."
+          "By default, it's set to Ratis RPC timeout value." +
+          "Setting this to -1 allows read to return immediately."


Since there are two limits, it may not "return immediately". How about below?

"Setting this to -1 to allow infinite lag."

szetszwo

+1 The change looks good.

ivandika3 · 2026-02-27T15:12:58Z

@szetszwo Thank you for the thorough reviews.

ivandika3 added 2 commits February 20, 2026 16:18

HDDS-14509. Allow client to choose the read consistency level

1933b2a

Revert unnecessary change

ff24470

ivandika3 requested review from symious, szetszwo and xichen01 February 20, 2026 09:54

Revert another unnecessary change

cc9e3eb

ivandika3 self-assigned this Feb 22, 2026

Revert some refactoring

7bdb5a8

Revert some refactoring

f3c254a

szetszwo reviewed Feb 23, 2026

View reviewed changes

ivandika3 added 3 commits February 24, 2026 11:49

Refine the ReadConsistency semantics

e6fd8ca

Use hasLeaseLagLimit and hasLeaseTimeMsLimit, allow infinite lag

5672288

Local lease for leader not ready should be handled like follower

94c8f76

szetszwo reviewed Feb 24, 2026

View reviewed changes

ivandika3 added 7 commits February 25, 2026 10:29

Change UNKNOWN_READ_CONSISTENCY to UNSPECIFIED

55797e5

Build default consistency hint once

e7f2248

Static for getRaftReadRequestType

f1bd4ad

Refine local lease server logic, change configuration

8e8f5cd

LEADER_AND_NOT_READY and NOT_LEADER handled similarly in LINEARIZABLE…

65b4833

…_ALLOW_FOLLOWER

Refactor OM read request processing

b8dabb2

Fix test, refine local lease, rename variables

29e044c

szetszwo reviewed Feb 25, 2026

View reviewed changes

ivandika3 added 3 commits February 26, 2026 17:16

Use READ_CONSISTENCY_UNSPECIFIED

5e58306

Refactor specifiedReadConsistency

4f62751

Update leaseTimeMs logic

09efae1

szetszwo reviewed Feb 26, 2026

View reviewed changes

ivandika3 added 2 commits February 27, 2026 17:51

Address comment

330685f

Test for local lease infinite lag

ad28253

Fix both config description

92154a1

szetszwo approved these changes Feb 27, 2026

View reviewed changes

ivandika3 marked this pull request as ready for review February 27, 2026 15:12

szetszwo merged commit 1edbb6d into apache:master Feb 27, 2026
101 of 102 checks passed

		if (!omRequest.hasReadConsistencyHint() \|\| !omRequest.getReadConsistencyHint().hasReadConsistency() \|\|
		omRequest.getReadConsistencyHint().getReadConsistency() == ReadConsistencyProto.UNSPECIFIED) {

Conversation

ivandika3 commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

szetszwo commented Feb 21, 2026

Uh oh!

ivandika3 commented Feb 22, 2026

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivandika3 Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ivandika3 commented Feb 20, 2026 •

edited

Loading

szetszwo Feb 23, 2026 •

edited

Loading

szetszwo Feb 23, 2026 •

edited

Loading

ivandika3 Feb 24, 2026 •

edited

Loading