-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 2.6.3
-
Component/s: Index Maintenance, Storage
-
None
-
None
-
3
-
None
-
None
-
None
-
None
-
None
-
None
We're having a problem with one of our shards - after running a failed repair process. subsequent repair fail as well, with the above error.
also , we can't query one of our collections with the same error.
we would like to find a way to recover from this situation, we didn't see any documentation for this error in the other issues.
here is the relevant log
2015-07-23T09:53:25.201+0300 [FileAllocator] allocating new datafile /var/lib/mongodb/_tmp_repairDatabase_0/cuckoo.55, filling with zeroes... 2015-07-23T09:53:25.214+0300 [FileAllocator] done allocating datafile /var/lib/mongodb/_tmp_repairDatabase_0/cuckoo.55, size: 2047MB, took 0.011 secs 2015-07-23T09:53:31.060+0300 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms 2015-07-23T09:53:35.851+0300 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms 2015-07-23T09:53:36.017+0300 [initandlisten] Assertion: 10320:BSONElement: bad type -31 2015-07-23T09:53:36.056+0300 [initandlisten] cuckoo.fs.chunks 0x11da801 0x117c9f9 0x11614b6 0x76fe6a 0xb6eb54 0xb708d0 0xb71541 0xb739e7 0xb69773 0x8e7f74 0x8b20dc 0xdf979e 0x76398e 0x76602c 0x76a7 1f 0x76affb 0x76b595 0x76b7b9 0x7f5ccee69ec5 0x7626e9 mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11da801] mongod(_ZN5mongo10logContextEPKc+0x159) [0x117c9f9] mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11614b6] mongod(_ZNK5mongo11BSONElement4sizeEv+0x1ca) [0x76fe6a] mongod(_ZNK5mongo19BtreeKeyGeneratorV118extractNextElementERKNS_7BSONObjES3_RPKcRb+0x1c4) [0xb6eb54] mongod(_ZNK5mongo19BtreeKeyGeneratorV120getKeysImplWithArrayESt6vectorIPKcSaIS3 _EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjEPSt3setIS9_NS_10BSONObjCmpESaIS9_EEjSB_+0x100) [0xb708d0] mongod(_ZNK5mongo19BtreeKeyGeneratorV111getKeysImplESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjEPSt3setIS9_NS_10BSONObjCmpESaIS9_EE+0x181) [0xb71541] mongod(_ZNK5mongo17BtreeKeyGenerator7getKeysERKNS_7BSONObjEPSt3setIS1_NS_10BSONObjCmpESaIS1_EE+0x2a7) [0xb739e7] mongod(_ZN5mongo9BtreeBulk6insertERKNS_7BSONObjERKNS_7DiskLocERKNS_19InsertDeleteOptionsEPlPKNS_23PregeneratedKeysOnIndexE+0x83) [0xb69773] mongod(_ZN5mongo15MultiIndexBlock6insertERKNS_7BSONObjERKNS_7DiskLocERKNS_19InsertDeleteOptionsE+0x94) [0x8e7f74] mongod(_ZN5mongo10Collection14insertDocumentERKNS_7BSONObjERNS_15MultiIndexBlockE+0x7c) [0x8b20dc] mongod(_ZN5mongo14repairDatabaseESsbb+0x234e) [0xdf979e] mongod(_ZN5mongo11doDBUpgradeERKSsPNS_14DataFileHeaderE+0x5e) [0x76398e] mongod() [0x76602c] mongod(_ZN5mongo14_initAndListenEi+0x5df) [0x76a71f] mongod(_ZN5mongo13initAndListenEi+0x1b) [0x76affb] mongod() [0x76b595] mongod(main+0x9) [0x76b7b9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f5ccee69ec5] mongod() [0x7626e9] 2015-07-23T09:53:36.301+0300 [initandlisten] cleaning up failed repair db: cuckoo path: /var/lib/mongodb/_tmp_repairDatabase_0 2015-07-23T09:53:39.737+0300 [initandlisten] exception in initAndListen: 10320 BSONElement: bad type -31, terminating 2015-07-23T09:53:39.739+0300 [initandlisten] dbexit: 2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to close listening sockets... 2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to flush diaglog... 2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to close sockets... 2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: waiting for fs preallocator... 2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: closing all files... 2015-07-23T09:53:40.282+0300 [initandlisten] closeAllFiles() finished 2015-07-23T09:53:40.282+0300 [initandlisten] shutdown: removing fs lock... 2015-07-23T09:53:40.283+0300 [initandlisten] dbexit: really exiting now