In FCBIS today, if we have failed while running commands on our sync source, we may denylist that node as sync source for 60s. This includes if we cannot open the backup cursor for a variety of reasons. One of those reasons includes there is already an open backup cursor on our sync source, since we only allow one backup cursor to be open at a time.

However, when we start multiple new nodes for FCBIS (including initiating a new set), it is possible that there are more nodes trying to initial sync than there are nodes available as sync source. As a result, FCBIS nodes could fail multiple attempts and end up denylisting all potential sync sources. As a result, the node will restart another attempt, and then end up failing to find any valid sync sources, which will use up another initial sync attempt. As mentioned earlier, this is especially common in initiating cases.

Acceptance Criteria

To prevent us from crashing nodes unnecessarily, we should consider checking if we have denylisted all nodes. If that is the reason why we cannot select a sync source, then do not use up an initial sync attempt; instead, we can block FCBIS from retrying another attempt until a node has been removed from the denylist.