We ran sharded mongo cluster on Mongo 4.0.13 on Ubuntu 18.04 x64 .

A single (non-sharded) mongo was initially added to cluster (srv211), later 3 other mongo servers were added and collections were moved to 3 new shards. srv211 shard was removed, all mongos services were restarted (shard/cfg instances were not).

But suddenly after several months of uptime, the cluster was blocked and we got the following error messages in one of our shard logs:

 

{{2019-12-30T19:35:18.079+0000 I SHARDING [conn123235] received splitChunk request: { splitChunk: "project_prod_user.player_maps", from: "project-db-3", keyPattern:

{ pers_id: "hashed" }

, epoch: ObjectId('5dafd8dc63ae2c18230f9dce'), shardVersion: [ Timestamp(10814, 0), ObjectId('5dafd8dc63ae2c18230f9dce') ], min: { pers_id: 4618768445950617740 }, max: { pers_id: 4620991399317296785 }, splitKeys: [ { pers_id: 4619791202844536983 }, { pers_id: 4620989830407096034 } ], $clusterTime: { clusterTime: Timestamp(1577734518, 5), signature:

{ hash: BinData(0, 4219F00DC9FE995B5CBBBC9B5E39EBFF35C51A10), keyId: 6750833113431015451 }

}, $configServerState: { opTime:

{ ts: Timestamp(1577734517, 30), t: 5 }

}, $db: "admin" }}}
2019-12-30T19:35:18.085+0000 I SHARDING [conn123235] distributed lock 'project_prod_user.player_maps' acquired for 'splitting chunk [{ pers_id: 4618768445950617740 }, { pers_id: 4620991399317296785 }) in project_prod_user.player_maps', ts : 5e0a5176dc25d0219069dd07
2019-12-30T19:35:18.136+0000 I SHARDING [conn123235] distributed lock with ts: 5e0a5176dc25d0219069dd07' unlocked.
2019-12-30T19:35:18.148+0000 I SHARDING [conn125292] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.148+0000 I SHARDING [conn125292] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.149+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.150+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.150+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.152+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.152+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.156+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.156+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.168+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.171+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.174+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found 

 

I've examined config database, but found no mentions of srv211 in db.chunks or db.shards.

Is it a known issue?