[SERVER-51067] Chunk move failed :: caused by :: IndexNotFound: can't find index with prefix Created: 19/Sep/20  Updated: 27/Oct/23  Resolved: 30/Sep/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: zhenchuan ding Assignee: Dmitry Agranat
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

我给我们的mongo 集群 新加了一个分片,但是在老得shard上 发现日志有报错,并且数据基本上没有均衡,均衡的功能是打开的,报错日志如下:

2020-09-18T12:37:17.589+0000 W SHARDING [conn1187835] Chunk move failed :: caused by :: IndexNotFound: can't find index with prefix { _id: "hashed" } in storeCurrentLocs for 202jgkbx.user
2020-09-18T12:37:44.703+0000 W SHARDING [conn1187835] Chunk move failed :: caused by :: IndexNotFound: can't find index with prefix { _id: "hashed" } in storeCurrentLocs for 232cf15a.user
2020-09-18T12:37:55.434+0000 W SHARDING [conn1187835] Chunk move failed :: caused by :: IndexNotFound: can't find index with prefix { device_id: 1.0 } in storeCurrentLocs for 566d12f9.event
2020-09-18T12:38:16.381+0000 W SHARDING [conn1187835] Chunk move failed :: caused by :: IndexNotFound: can't find index with prefix { _id: "hashed" } in storeCurrentLocs for 453f92cc.user
2020-09-18T12:38:28.460+0000 W SHARDING [conn1187835] Chunk move failed :: caused by :: IndexNotFound: can't find index with prefix { _id: "hashed" } in storeCurrentLocs for 69iqa0j0.user



 Comments   
Comment by Dmitry Agranat [ 30/Sep/20 ]

Hi zhenchuan.ding@ptmind.com, thank you for clarification about a bug in your application.

As for the data balancing and further tuning, I encourage you to ask our community by posting on the MongoDB Developer Community Forums or on Stack Overflow with the mongodb tag.

Regards,
Dima

Comment by zhenchuan ding [ 25/Sep/20 ]

The version we used was 4.2.8. We started with two shards, and then I added three Shards. However, the disk usage of the previous two shards is still growing, and the speed of data migration is particularly slow. At present, the data is balanced for two days, and the disk usage of the new shards is only one third of the previous Shards. Our total data is about 500g. Is this normal

Comment by zhenchuan ding [ 25/Sep/20 ]

 
Hello, I'm very sorry. The fault is caused by a bug in our program, which has been solved. thank you

Comment by Dmitry Agranat [ 24/Sep/20 ]

Hi zhenchuan.ding@ptmind.com,

Could you please mention the current MongoDB version?

Thanks,
Dima

Comment by zhenchuan ding [ 22/Sep/20 ]

2020-09-22T09:59:50.072+0000 W SHARDING [conn394] Chunk move failed :: caused by :: IndexNotFound: can't find index with prefix \{ _id: "hashed" } in storeCurrentLocs for 305xxfg4.user
 2020-09-22T09:59:50.075+0000 I COMMAND [conn394] command admin.$cmd command: moveChunk { moveChunk: "305xxfg4.user", shardVersion: [ Timestamp(1, 0), ObjectId('5eb7deb337ece56aa995afd4') ], epoch: ObjectId('5eb7deb337ece56aa995afd4'), configdb: "myset/172.19.11.117:27000,172.19.11.118:27000,172.19.11.119:27000", fromShard: "shard1", toShard: "shard3", min:
 
{ _id: MinKey }
 
, max: \{ _id: -4611686018427387902 }, maxChunkSizeBytes: 67108864, waitForDelete: false, takeDistLock: false, $clusterTime: { clusterTime: Timestamp(1600768789, 22), signature:
 
{ hash: BinData(0, A3F335CF2D329FA4FC109D5593BEDF213DE12FD8), keyId: 6841756763633483777 }
 
}, $configServerState: { opTime:
 
{ ts: Timestamp(1600768789, 22), t: 12 }
 
}, $db: "admin" } numYields:2 ok:0 errMsg:"can't find index with prefix \{ _id: \"hashed\" } in storeCurrentLocs for 305xxfg4.user" errName:IndexNotFound errCode:27 reslen:472 locks:{ ParallelBatchWriterMode: { acquireCount:
 
{ r: 10 }
 
}, ReplicationStateTransition: { acquireCount:
 
{ w: 466 }
 
}, Global: { acquireCount:
 
{ r: 463, w: 3 }
 
}, Database: { acquireCount:
 
{ r: 10, w: 3 }
 
}, Collection: { acquireCount:
 
{ r: 8, w: 2 }
 
}, Mutex: { acquireCount:
 
{ r: 12, W: 2 }
 
}, oplog: { acquireCount:
 
{ r: 2, w: 1 }
 
} } flowControl:\{ acquireCount: 2 } storage:{ data:
 
{ bytesRead: 4763007, timeReadingMicros: 13948 }
 
} protocol:op_msg 615ms

 

 

 

This is the latest log

Comment by zhenchuan ding [ 21/Sep/20 ]

2020-09-21T12:18:48.395+0000 I SHARDING [ChunkSplitter-13221] Unable to auto-split chunk [\{ _id: 4611686018427387902 }, \{ _id: MaxKey }) in namespace 3ff06b18.user :: caused by :: IndexNotFound: couldn't find index over splitting key \{ _id: "hashed" }
2020-09-21T12:18:48.488+0000 I SHARDING [ChunkSplitter-13219] Unable to auto-split chunk [\{ _id: 0 }, \{ _id: 4611686018427387902 }) in namespace 549ce0ff.user :: caused by :: IndexNotFound: couldn't find index over splitting key \{ _id: "hashed" }
2020-09-21T12:18:48.533+0000 I SHARDING [ChunkSplitter-13221] Unable to auto-split chunk [\{ _id: 4611686018427387902 }, \{ _id: MaxKey }) in namespace 549ce0ff.user :: caused by :: IndexNotFound: couldn't find index over splitting key \{ _id: "hashed" }
2020-09-21T12:18:49.039+0000 I SHARDING [ChunkSplitter-13219] Unable to auto-split chunk [\{ device_id: "38CaWOcv8/kZYg3U1rLoTw" }, \{ device_id: "5h3EwrVKsSqgDj5mWiX9Qw" }) in namespace 566d12f9.event :: caused by :: IndexNotFound: couldn't find index over splitting key \{ device_id: 1.0 }
2020-09-21T12:18:49.207+0000 I SHARDING [ChunkSplitter-13221] Unable to auto-split chunk [\{ _id: 0 }, \{ _id: 4611686018427387902 }) in namespace 3ff06b18.user :: caused by :: IndexNotFound: couldn't find index over splitting key \{ _id: "hashed" }

Generated at Thu Feb 08 05:24:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.