I've created a test case (see attachement) that can create any number of tables that share the same table id. This may be resolved after SPM-4546, but it's an issue on develop now.

How to repro this

• On the leader node, after final checkpoint, create N tables
• Close leader connection with skip_checkpoint

  • self.close_conn(config='debug=(skip_checkpoint=true)')
    

• • Reason to skip checkpoint is to simulate a crash or non-checkpoint stepdown.
• Follower step up, and then create another N tables
• ID Conflict here: both the old N tables and new N tables share the same ID
• And it can survive for several checkpoints until it's down
  • Another side issue here is because both tables share the same id (with different uri), so if they also share the same key, then the node may silently crash in reconcile when they validate the HS storage which use btree id as the link to live table.
• Next time when the leader step up it will crash.
  • Because it will then fail the boot verify rule of conflict id
  • But because it survives the whole life time of previous leader period, so there's may already lots of data (and may conflicts in HS too).

Why it's possible

The reason for this is we're using the connection level local variable next_file_id to generate next table id, however the cross-node sync of this variable is utilising the checkpoint metadata (not the uri but connection level) and the update of this variable is at the time point when we create new schema. So even step-up with latest checkpoint still won't solve this problem.  

test_id_collision_natural.py