[SERVER-19534] Assertion: 10320:BSONElement: bad type -31 Created: 23/Jul/15  Updated: 27/Jul/15  Resolved: 27/Jul/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Storage
Affects Version/s: 2.6.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nitzan Carmel Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

We're having a problem with one of our shards - after running a failed repair process. subsequent repair fail as well, with the above error.

also , we can't query one of our collections with the same error.

we would like to find a way to recover from this situation, we didn't see any documentation for this error in the other issues.

here is the relevant log

2015-07-23T09:53:25.201+0300 [FileAllocator] allocating new datafile /var/lib/mongodb/_tmp_repairDatabase_0/cuckoo.55, filling with zeroes...
2015-07-23T09:53:25.214+0300 [FileAllocator] done allocating datafile /var/lib/mongodb/_tmp_repairDatabase_0/cuckoo.55, size: 2047MB,  took 0.011 secs
2015-07-23T09:53:31.060+0300 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms
2015-07-23T09:53:35.851+0300 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms
2015-07-23T09:53:36.017+0300 [initandlisten] Assertion: 10320:BSONElement: bad type -31
2015-07-23T09:53:36.056+0300 [initandlisten] cuckoo.fs.chunks 0x11da801 0x117c9f9 0x11614b6 0x76fe6a 0xb6eb54 0xb708d0 0xb71541 0xb739e7 0xb69773 0x8e7f74 0x8b20dc 0xdf979e 0x76398e 0x76602c 0x76a7
1f 0x76affb 0x76b595 0x76b7b9 0x7f5ccee69ec5 0x7626e9
 mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11da801]
 mongod(_ZN5mongo10logContextEPKc+0x159) [0x117c9f9]
 mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11614b6]
 mongod(_ZNK5mongo11BSONElement4sizeEv+0x1ca) [0x76fe6a]
 mongod(_ZNK5mongo19BtreeKeyGeneratorV118extractNextElementERKNS_7BSONObjES3_RPKcRb+0x1c4) [0xb6eb54]
 mongod(_ZNK5mongo19BtreeKeyGeneratorV120getKeysImplWithArrayESt6vectorIPKcSaIS3
_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjEPSt3setIS9_NS_10BSONObjCmpESaIS9_EEjSB_+0x100) [0xb708d0]
 mongod(_ZNK5mongo19BtreeKeyGeneratorV111getKeysImplESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjEPSt3setIS9_NS_10BSONObjCmpESaIS9_EE+0x181) [0xb71541]
 mongod(_ZNK5mongo17BtreeKeyGenerator7getKeysERKNS_7BSONObjEPSt3setIS1_NS_10BSONObjCmpESaIS1_EE+0x2a7) [0xb739e7]
 mongod(_ZN5mongo9BtreeBulk6insertERKNS_7BSONObjERKNS_7DiskLocERKNS_19InsertDeleteOptionsEPlPKNS_23PregeneratedKeysOnIndexE+0x83) [0xb69773]
 mongod(_ZN5mongo15MultiIndexBlock6insertERKNS_7BSONObjERKNS_7DiskLocERKNS_19InsertDeleteOptionsE+0x94) [0x8e7f74]
 mongod(_ZN5mongo10Collection14insertDocumentERKNS_7BSONObjERNS_15MultiIndexBlockE+0x7c) [0x8b20dc]
 mongod(_ZN5mongo14repairDatabaseESsbb+0x234e) [0xdf979e]
 mongod(_ZN5mongo11doDBUpgradeERKSsPNS_14DataFileHeaderE+0x5e) [0x76398e]
 mongod() [0x76602c]
 mongod(_ZN5mongo14_initAndListenEi+0x5df) [0x76a71f]
 mongod(_ZN5mongo13initAndListenEi+0x1b) [0x76affb]
 mongod() [0x76b595]
 mongod(main+0x9) [0x76b7b9]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f5ccee69ec5]
 mongod() [0x7626e9]
2015-07-23T09:53:36.301+0300 [initandlisten] cleaning up failed repair db: cuckoo path: /var/lib/mongodb/_tmp_repairDatabase_0
2015-07-23T09:53:39.737+0300 [initandlisten] exception in initAndListen: 10320 BSONElement: bad type -31, terminating
2015-07-23T09:53:39.739+0300 [initandlisten] dbexit:
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to close listening sockets...
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to flush diaglog...
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to close sockets...
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: waiting for fs preallocator...
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: closing all files...
2015-07-23T09:53:40.282+0300 [initandlisten] closeAllFiles() finished
2015-07-23T09:53:40.282+0300 [initandlisten] shutdown: removing fs lock...
2015-07-23T09:53:40.283+0300 [initandlisten] dbexit: really exiting now



 Comments   
Comment by Sam Kleinman (Inactive) [ 27/Jul/15 ]

This kind of error typically indicates some sort of issue with data corruption, which is often caused by problems with the underlying storage device, file system or network connection.

You can use this procedure to resync this member from a healthy member of the replica set.

For additional support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
sam

Generated at Thu Feb 08 03:51:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.