Details
-
Bug
-
Resolution: Done
-
Major - P3
-
None
-
2.6.3
-
None
Description
We're having a problem with one of our shards - after running a failed repair process. subsequent repair fail as well, with the above error.
also , we can't query one of our collections with the same error.
we would like to find a way to recover from this situation, we didn't see any documentation for this error in the other issues.
here is the relevant log
2015-07-23T09:53:25.201+0300 [FileAllocator] allocating new datafile /var/lib/mongodb/_tmp_repairDatabase_0/cuckoo.55, filling with zeroes...
|
2015-07-23T09:53:25.214+0300 [FileAllocator] done allocating datafile /var/lib/mongodb/_tmp_repairDatabase_0/cuckoo.55, size: 2047MB, took 0.011 secs
|
2015-07-23T09:53:31.060+0300 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms
|
2015-07-23T09:53:35.851+0300 [initandlisten] warning Listener::getElapsedTimeMillis returning 0ms
|
2015-07-23T09:53:36.017+0300 [initandlisten] Assertion: 10320:BSONElement: bad type -31
|
2015-07-23T09:53:36.056+0300 [initandlisten] cuckoo.fs.chunks 0x11da801 0x117c9f9 0x11614b6 0x76fe6a 0xb6eb54 0xb708d0 0xb71541 0xb739e7 0xb69773 0x8e7f74 0x8b20dc 0xdf979e 0x76398e 0x76602c 0x76a7
|
1f 0x76affb 0x76b595 0x76b7b9 0x7f5ccee69ec5 0x7626e9
|
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11da801]
|
mongod(_ZN5mongo10logContextEPKc+0x159) [0x117c9f9]
|
mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11614b6]
|
mongod(_ZNK5mongo11BSONElement4sizeEv+0x1ca) [0x76fe6a]
|
mongod(_ZNK5mongo19BtreeKeyGeneratorV118extractNextElementERKNS_7BSONObjES3_RPKcRb+0x1c4) [0xb6eb54]
|
mongod(_ZNK5mongo19BtreeKeyGeneratorV120getKeysImplWithArrayESt6vectorIPKcSaIS3
|
_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjEPSt3setIS9_NS_10BSONObjCmpESaIS9_EEjSB_+0x100) [0xb708d0]
|
mongod(_ZNK5mongo19BtreeKeyGeneratorV111getKeysImplESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjEPSt3setIS9_NS_10BSONObjCmpESaIS9_EE+0x181) [0xb71541]
|
mongod(_ZNK5mongo17BtreeKeyGenerator7getKeysERKNS_7BSONObjEPSt3setIS1_NS_10BSONObjCmpESaIS1_EE+0x2a7) [0xb739e7]
|
mongod(_ZN5mongo9BtreeBulk6insertERKNS_7BSONObjERKNS_7DiskLocERKNS_19InsertDeleteOptionsEPlPKNS_23PregeneratedKeysOnIndexE+0x83) [0xb69773]
|
mongod(_ZN5mongo15MultiIndexBlock6insertERKNS_7BSONObjERKNS_7DiskLocERKNS_19InsertDeleteOptionsE+0x94) [0x8e7f74]
|
mongod(_ZN5mongo10Collection14insertDocumentERKNS_7BSONObjERNS_15MultiIndexBlockE+0x7c) [0x8b20dc]
|
mongod(_ZN5mongo14repairDatabaseESsbb+0x234e) [0xdf979e]
|
mongod(_ZN5mongo11doDBUpgradeERKSsPNS_14DataFileHeaderE+0x5e) [0x76398e]
|
mongod() [0x76602c]
|
mongod(_ZN5mongo14_initAndListenEi+0x5df) [0x76a71f]
|
mongod(_ZN5mongo13initAndListenEi+0x1b) [0x76affb]
|
mongod() [0x76b595]
|
mongod(main+0x9) [0x76b7b9]
|
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f5ccee69ec5]
|
mongod() [0x7626e9]
|
2015-07-23T09:53:36.301+0300 [initandlisten] cleaning up failed repair db: cuckoo path: /var/lib/mongodb/_tmp_repairDatabase_0
|
2015-07-23T09:53:39.737+0300 [initandlisten] exception in initAndListen: 10320 BSONElement: bad type -31, terminating
|
2015-07-23T09:53:39.739+0300 [initandlisten] dbexit:
|
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to close listening sockets...
|
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to flush diaglog...
|
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: going to close sockets...
|
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: waiting for fs preallocator...
|
2015-07-23T09:53:39.740+0300 [initandlisten] shutdown: closing all files...
|
2015-07-23T09:53:40.282+0300 [initandlisten] closeAllFiles() finished
|
2015-07-23T09:53:40.282+0300 [initandlisten] shutdown: removing fs lock...
|
2015-07-23T09:53:40.283+0300 [initandlisten] dbexit: really exiting now
|