Failover support

[michalszynkiewicz]

We had a few asks for failover support in Stork.
Currently, we stand at the position that Stork should only provide service instances and not be involved in making calls, but maybe we should revisit it?

Currently, the way to solve it is to use a failure aware load balancer, such as least-response-time and combine it e.g. with MicroProfile Fault Tolerance @Retry annotation.

[michalszynkiewicz]

CC @mswiderski @vsevel

[mswiderski]

just linking here the issue I created some time ago that has some insights #147 I guess we should close on or the other as they seem like a duplicates of each other

[michalszynkiewicz]

@mswiderski I wanted a place to specifically discuss failover

[vsevel]

Many years ago all of our EAPs were deployed as 2-nodes clusters, without a load balancer in front.
Additionally, we also developed a proprietary protocol to allow ejb invocations over http (for our EAP5 to EAP6 migration).
We ended up having to develop our own client side load balancing functionality in our EAP servers, when calling remote endpoints (other EAP servers) for these 2 protocols.

Here is a description of the approach we took, which hopefully will help the discussion.

Let's say that for the remote service we have 3 urls: http://one:8080, http://two:8080, http://three:8080.
Each time a service needs to be called we assemble a list of all urls ordered by eligibility and load balancing priorities.
When everything is up, for the first call we return 1-2-3, then 2-3-1 on the second call, ...
If a call fails with a retriable error (more on that later), we record the error (with a timestamp attached to the service instance) and we continue on the list.
So if 2 returns a ConnectException, we will mark 2 as failed then try to execute the call on 3, then on 1 if 3 fails as well.
Let's say say 2 failed but 3 succeeded.
Next time we will assemble the list a little bit differently. We will start by creating the usual list 1-2-3 (assuming the last call was done on 2-3-1), but we separate this list in 2 sublists: instances with no failures and instances with failures.
so we would create the ok list with 1-3 and the nok list with just 2, then reassemble a complete list with ok+nok, so in that case: 1-3-2.
let's say that the call on 1 succeeds, next call we will have initial list=2-3-1, ok=3-1 and nok=2, and final list=3-1-2
most of the time we will be cycling on the good instances, and try out the nok instances only if all good instances have failed.

this is an important property: we always keep the nok instances in the list as a last resort.

a nok instance is good again if:

• it had a successful call (that is the situation where all good instances failed, and we used a nok instance, which happened to be good actually),
• or the last failure was more than 60 secs ago.

continuing from the previous example, we had: initial list=2-3-1, ok=3-1 and nok=2, and final list=3-1-2
next call we have initial list=3-1-2, ok=3-1 and nok=2, and final list=3-1-2 (note that a weakness of the algorithm is that in that case 3 will be used for 2 subsequent calls).
60 seconds pass by.
next call we have initial list=1-2-3, ok=1-2-3 and nok=[], final list=1-2-3 (2 is considered good again).

an important aspect of the whole mechanism is recognizing when you can retry, and when you must not. this actually depends on the protocol used.

for rest, we used the idempotency property of the http verbs defined in the spec.
idempotent calls would be retried only if we were facing an IOException that was not a com.fasterxml.jackson.core.JsonProcessingException.

for ejb/http, since all calls are going through a POST, we created a dedicated @Idempotent annotation that we would add on the java methods of the interface describing the remote endpoint.
any @Idempotent annotated method that failed with an IOException would be retried.

in addition to these protocol specific conditions, we would recognize certain situations where we knew for sure that the call did not happen (including exceptions specific to libs we were using underneath):

• java: java.net.ConnectException and java.net.UnknownHostException,
• apache http:org.apache.http.con.ConnectTimeoutException and org.apache.http.con.HttpHostConnectException,
• resteasy: org.jboss.resteasy.client.exception.ResteasyHttpHostConnectException

We would retry also on 503.

the fact that client side load balancing handled a number of retries automatically relieved the developers from having to deal with this at the app resilience level (e.g. MP Fault Tolerance). so for instance developers did not have to add @Retry annotations to all of their @GET endpoints. And they did not have to add @Retry(retryOn = "ConnectException") to all of their methods.

they knew that the obvious errors would be handled underneath, and they needed to add fault tolerance only for the extra situations that needed some app knowledge.

the other benefit was they did not have to know how many retries to do to make sure they touched all nodes in the cluster, which they could not know at the app level. the client side load balancing knows how many instances there are. so in a 3 nodes cluster, there is no need to retry 50 times an idempotent method. 3 is enough. Similarly, if we know we have a 3 nodes cluster, it is worth trying 3 times rather than 2. The application may still need/want to do some extra retries if it suspects the issue is not with the endpoint itself but with one of its dependencies (e.g. a db). but that is a different level of retry.

part of this designed has been influenced by the retry mechanism in apache http client:

• HttpRequestRetryExec
• DefaultHttpRequestRetryStrategy

the algorithm has been running for roughly 10 years, and started way before there was a resiliency libs available. it provided us with a lot of stability, even through crashes, restarts and redeploys. as we are moving to k8s, the need for client side load balancing will probably diminish, but we still have to deal with our legacy infrastructure, so this component is going to stay critical for many more years I suspect.

I think stork is very well positioned for us considering our situation and the investment we are making in Quarkus. I am very excited about using it.
to be really effective it needs to go farther in its ability to select several eligible instances for a single call and provide automatic retries when it can.

happy to discuss it further.

[mswiderski]

they knew that the obvious errors would be handled underneath, and they needed to add fault tolerance only for the extra situations that needed some app knowledge.

this is the culprit from my point of view. Load balancer should automatically handle the connectivity errors based on the state of instances it is aware of. And only propagate errors that are not related to connectivity as it cannot figure out on its own if particular error should be retried or not.

[vsevel]

Load balancer should automatically handle the connectivity errors based on the state of instances it is aware of.

we have to recognize that awareness is a weak property here. there are instances that you think are good, but are not. and some instances that you think are bad, but are not.
that is why in our design our load balancer returns a list of instances and not just one. and based on protocol/endpoint retriability, it may go through the entire list.

as it cannot figure out on its own if particular error should be retried or not.

it can actually in some cases. the obvious example is idempotent http verbs in rest, for instance in apache httpclient the retryRequest() method relying on the isIdempotent() definition.

in some cases it may not be appropriate for a particular protocol. one for instance could argue that a 503 should be reported back to the app layer and not retrioed. in our impl we decided that even 503 were retriable.

in a flexible design, there should be a retry strategy interface (similar in spirit to HttpRequestRetryStrategy) with a default implementation. some context on the type of call we are making should be passed to this strategy (at the very least the http verb, but ideally some reference on the java business method invoked on the proxy, if any, so that we can look at the annotations), so that the custom retry strategy can make intelligent decisions.

[mswiderski]

we have to recognize that awareness is a weak property here. there are instances that you think are good, but are not. and some instances that you think are bad, but are not.

agreed but there must be something to start with and I believe relying on connection related errors is a good start and certainly not that it has to end there.

it can actually in some cases. the obvious example is idempotent http verbs in rest, for instance in apache httpclient the retryRequest() method relying on the isIdempotent() definition.

alright, it can find certain things to rely on but again (and as you said later) it can be hidden into various types of retry strategy implementations with reasonable default if needed.

[vsevel]

oh yes sure.
I was certainly not saying that there is only one truth, and everything needed to be implemented to be correct.
but knowing the use cases you want to support will have an impact on the spi interfaces and the overall design.

grpc also added support for idempotent operations a few years ago using a method level idempotency option.

as discussed here: Calls that are marked as idempotent may be sent multiple times.

this means that client side load balancing (and failover) library would be well inclined to retry idempotent calls if it thought the call may succeed on other service instances.

if an api designer specified that an operation was idempotent, a client user should not have to re-state that the service should be retried at the Fault Tolerance level.

[michalszynkiewicz]

for me, it's outside of the scope of Stork.
The client library (that uses Stork) can choose to ask for a next service instance and do a retry, and if a load balancer keeps track of errors, it will most likely return a different instance.

Why do you think it should be handled by the load balancer?

[mswiderski]

Why do you think it should be handled by the load balancer?

mainly because the load balancer is completely hidden from client code. In my case I use MP Rest Client (with quarkus rest easy reactive client) so I have no access to the load balancer though I have configured multiple endpoints to make sure they will be tried in case of failures related to connectivity.

[michalszynkiewicz]

But there's also the client library. Which knows better the concurrency model (knows which thread/event loop should handle a retry), has the knowledge which operations are idempotent, etc.

@vsevel can the retry mechanism you described be also implemented as follows:

• a load balancer marks a service instance as blocked for some period of time after a failure
• the client is configured to retry many times (many more than we expect instances), unless NoAcceptableServiceInstanceFoundException is thrown - this is what LBs are meant to throw when all instances are deemed unavailable

The difference between this and iterating over a list of all service instances is that it will take into account failures from many calls, not only the current one. The situation of having many instances down is unlikely, they would probably be removed on the next refresh of service discovery. Creating an ordered list of service instances may be inefficient, depending on the load balancing strategy.

[vsevel]

Part of the discussion is deciding if your load balancer is acting as an intermediate between the client and the service (that is the traditional definition), or a name resolver (you give me a name and I give you an address you can call).
it seems that you are doing a little bit of both.
your load balancer is not an intermediate.
your default round robin load balancer looks almost like a round robin DNS, except that instead of providing a FQDN and getting a IP back, you give an abstract service name and you get a name (or ip) back.
once your load balancer has given the host:port back, it gets out of the loop.
you do implement some strategies in terms smart load distribution (not really a failover, nor health checks, but the least-response-time will favor instances that work well based on client feedback). so that is where I say it is a little bit of both.

The proprietary implementation I did, was closer to a client side load balancer (with no health checks), since it would serve as an intermediate between the client and the service, and was able to make informed failover decisions, because it was topology aware, and used client provided strategies to help figure out retriability for different protocol.

It is somehow similar to what the apache http client is doing with HttpRequestRetryExec, but limited to a single host. In my case I took what they were doing, and applied to a situation where the retries could be done on different hosts, rather than on the same one.

I looked at Ribbon. It seems to be what I would call client side load balancer since they would handle the call, and support retry options. check DefaultLoadBalancerRetryHandler and the different retry options such as MaxAutoRetries, MaxAutoRetriesNextServer, OkToRetryOnAllOperations, RequestSpecificRetryOn in CommonClientConfigKey.
Apparently there is also a retryableStatusCodes in spring cloud netflix as well.

I am not a ribbon expert, but the healthchecks seem to be provided by eureka (i.e. discovery), rather than ribbon itself.

I am wondering if stork is closer to consul from that perspective where the name you resolve will point to a real service that is supposed to be up, according to consul's healthchecks. But if that is not the case, that is the responsibility of the client to ask for new address and retry.

The issue with that approach is that each client (e.g. grpc, rest client) needs to implement auto retry on common situations (e.g. ConnectException), and since they are not topology aware, they can only guess how many tries they can/should make. Also, since the load balancer component will round robin across all addresses based on all calls happening in parallel, a retry may actually land on the same server.

I am not saying this is wrong. However it must be understood that the retriability needs to be moved upward in the clients (e.g. rest client, grpc). So for instance for grpc, some code will have to figure out that the first try failed with a retriable error, in a situation where a second call may actually pass (it would have to know that a stork:// url means that there probably multiple instances behind), and as a result will need to re-fetch a host from the load balancer, and re-execute the code.

As stated previously, and easy way to do this is to push that back to the application developer by asking him to use MP Fault Tolerance for this. But that is a burden that he/she should have not to endure since we know that some errors are always retriable, and some of the protocols have already defined rules about retriability (e.g. idempotent verbs for rest, idempotency annotation for grpc). So the developer should not have to re-state it.

So where does it leave us? With the current approach, I guess you need to make sure all client stacks implement auto-retry, some of logic being duplicated (e.g. reacting to connection errors for instance).

In my experience, we took a different approach where this concern was handled centrally.

[vsevel]

When we wrote our client side load balancing component, we also wrote a dedicated test suite to validate the behavior.
We would run a client application (with the client side load balancing component) and a backend.
The client application would call the backend application using a mix of short and long calls (Thread.sleep() on the server side) with a high throughput (many threads, each thread repeats calls as fast as possible).
The backend would be deployed as a cluster of 2 nodes.
On the server side, we would take one of those nodes and play 2 types of incidents:

• crash: kill -9 on the process
• graceful shutdown: let in flight calls finish (up to a timeout), and redirect new calls to the other server, stop the app when all inflight calls have finished

After the backend had been stopped, we would restart it, make sure load balancing had resumed normally on both server nodes, and play another incident.

The client side would make calls on 2 types of endpoints:

• idempotent
• non idempotent

The client application was not using any resilience lib (e.g. resilience4j, MP fault tolerance, ...). All retries were done transparently by the client side load balancing component.

We would let the test run for tenths of minutes (hundreds of thousands calls), and assess at the end the following conditions for success:

For crash situations:

• client side:
  • idempotent calls have all succeeded failures == 0
  • non idempotent calls may fail failures <= 1
• server side:
  • idempotent calls have been executed at least once exec >= 1
  • non idempotent calls may be executed at most once exec <= 1

For graceful shutdown situations:

• client side: all calls have succeeded (idempotent and non idempotent calls) failures == 0
• server side: all calls have executed exactly once exec == 1

Watching the type of errors during the tests (for instance the errors the client would receive while the backend was restarting), would allow to categorize them in either retriable and non retriable, depending on the protocol, and adjust the failure conditions on the client side for the automatic retries.

Eventually we got complete coverage on all possible errors.

I finished integrating stork in my pilot application.
No surprise, but the client application is very sensitive to the stability of the invoked remote services.
Since stork does not have a health check on those services, and no automatic retries, there will be guaranteed occasional failures on the client side, forcing using a resilience mechanism at the app level.
As previously discussed this will not work for us at scale (a few hundreds apps, thousands of endpoints). We have a little bit of time, but I am not sure how to approach this. A solution would be to plug a custom invoker in resteasy and migrate my original client side load balancing component, which I would like to avoid as much as possible, since stork in providing a solution in this exact area.

Please let me know how you see stork going forward.

[vsevel]

Interesting reading from the grpc project: Transparent Retries

Grpc already distinguishes retries that it will do automatically (aka transparent retries) from retries that are governed by the retry policy. 3 types of failures are defined:

• The RPC never leaves the client.
• The RPC reaches the server, but has never been seen by the server application logic.
• The RPC is seen by the server application logic, and fails.
  The 2 first ones are handled by transparent retries, the last one is handled by the retry policy, for instance on retriable code UNAVAILABLE(14), as demonstrated by this example.

In the first case, the call is retried multiple times until the deadlines passes.
In the 2nd case, the call is retried only once, then handled by the retry policy.

So for grpc, this may already work well.

I looked at the resteasy reactive ClientSendRequestHandler, and I did not see anything like this.