System split operations running in FATE could cause problem for user FATE operations

[keith-turner]

In #3382 system initiated tablet splits were moved to the manager and are now driven by a FATE operation. These fate operations could cause the following problems.

• Use up too much ZK memory in the case that lots of tablets try to split.
• Take FATE threads from user actions initiated in the API, delaying those operations.
• Cuases write pressure on ZK.

[keith-turner]

One possible solution discussed with @dlmarion was to create a separate FATE instantiation for system operations. This instantiation would have its own storage area in ZK and its own threads. It would also offer the ability to reject FATE operations when there are certain number active, this is something that FATE does not currently support. This would address the memory and execution concerns.

[keith-turner]

#3559 addresses some of the problems mentioned in the solution above. The solution in #3559 does not explicitly address having multiple instantiations of FATE, but it could support that. The solution mentioned in my previous comment does not address the problem of write pressure on ZK like #3559 does.

[dlmarion]

@keith-turner - Can this be closed now? User and System fate transactions now have their own Fate instance in the Manager, and @kevinrr888 is working on (or maybe completed at this time) dedicating threads to certain ops within a Fate instance.

[kevinrr888]

Should get confirmation from Keith, but I believe this can be closed now or maybe after #5301 is merged. Can link that this PR would close this.

As that PR is right now, it would give dedicated resources to SYSTEM_SPLIT ops, so SYSTEM_SPLIT will not interfere with any other operations