Same symptom. Different cause.
A local success bit while users still fail
Incomplete Recovery · CyberFort Grid
The Bad Failover
Failover reported success. Clients still cannot commit.
Client commits keep failing after the procedure that was supposed to restore them.
DEMONSTRATION SCENARIO
This is a demonstration scenario. It is not a live demonstration and does not report an executed result.
The situation
A data service has failed over. The procedure reports that it completed.
The failover bit is true. Clients cannot commit.
Why this is hard
Clients can still fail after failover because the path is wrong, because an upstream outage still needs time, or because only a health light went green. Those endings are different.
Running the failover again, or closing the incident on the success bit, should not count while commits fail.
What the operator can see
- SeenFailover bit true
- Open questionWhy commits fail
- At riskCommits keep failing
What V8Y must determine
V8Y must distinguish a finished failover from an early health bit and from an upstream fault that only needs time, and say what is still blocking client commits.
What may happen next
Client commits keep failing on the path clients are actually using.
What counts as recovery
Not recovery
Success is not a completed failover.
Recovery
Success is a client commit on the path the service was supposed to be using.
Observed results
No executed results are published for this thread.
What this evaluation would test
- Procedure success separated from the data path.
- Not the same false ending as a green health bit.
- The data path distinguished from the failover bit.
- Recovery judged on a client commit.
These are the conditions the evaluation would test. They are not reported results.
Operational basis
View operational basis and sources
Grounded in GitHub operator reports of failovers that completed while replicas stayed unattached, no primary was writable, or clients kept the wrong address. The facility setting here is synthetic.
GitHub. GitHub engineering report on the May 2023 availability event. 2023.
Related threads
Compare with Incomplete Recovery. There the misleading bit is health, not failover.
Discuss evaluating this situation
Does your environment face a similar problem?
The request keeps this mission and thread. It is not a claim that the demonstration has been validated in your environment.
The printable flyer is being revised and is not available for download.