-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 2.2.1
-
Component/s: Logging
-
Labels:None
-
Environment:Linux - Ubuntu Lucid
Linux db1e.songza.com 2.6.32-348-ec2 #54-Ubuntu SMP Fri Sep 7 11:21:33 UTC 2012 x86_64 GNU/Linux
-
ALL
We are getting what appear to be invalid bson documents in the system profile collection, which is making it rather difficult to analyze the results.
After enabling the profiler, these invalid documents will appear after a few minutes of run time. Dropping the capped collection and recreating it seems to have no effect as the invalid documents will be reinserted after a few minutes time. So far we have only seen this issue on our primary node. The secondary machines in our replica set do not appear to be affected. I don't recall ever getting this sort of error when we were running mongo 2.0. This is the first time since our upgrade to 2.2.1 that we have turned on the profiler.
Attempts to iterate through db.system.profile in the mongo shell result in the following assertion:
Wed Dec 12 16:16:59 Assertion: 10320:BSONElement: bad type -128 0x6073f1 0x5d1aa9 0x4b0d98 0x5c17a6 0x6b3f35 0x6b6a2c 0x69be0a 0x6aab74 0x695ebe 0x6aa5ac 0x69c025 0x6aa13f 0x668e46 0x668ec2 0x66a2ce 0x5cbcc4 0x4a47a5 0x4a67e6 0x7feb2ccf9c4d 0x49f669 mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x6073f1] mongo(_ZN5mongo11msgassertedEiPKc+0x99) [0x5d1aa9] mongo(_ZNK5mongo11BSONElement4sizeEv+0x1d8) [0x4b0d98] mongo(_ZN5mongo16resolveBSONFieldEP9JSContextP8JSObjectljPS3_+0x146) [0x5c17a6] mongo(js_LookupPropertyWithFlags+0x3f5) [0x6b3f35] mongo(js_GetProperty+0x7c) [0x6b6a2c] mongo(js_Interpret+0x10ea) [0x69be0a] mongo(js_Invoke+0x9d4) [0x6aab74] mongo() [0x695ebe] mongo(js_Invoke+0x40c) [0x6aa5ac] mongo(js_Interpret+0x1305) [0x69c025] mongo(js_Execute+0x36f) [0x6aa13f] mongo(JS_EvaluateUCScriptForPrincipals+0x66) [0x668e46] mongo(JS_EvaluateUCScript+0x22) [0x668ec2] mongo(JS_EvaluateScript+0x6e) [0x66a2ce] mongo(_ZN5mongo7SMScope4execERKNS_10StringDataERKSsbbbi+0x144) [0x5cbcc4] mongo(_Z5_mainiPPc+0x2455) [0x4a47a5] mongo(main+0x26) [0x4a67e6] /lib/libc.so.6(__libc_start_main+0xfd) [0x7feb2ccf9c4d] mongo(__gxx_personality_v0+0x2a1) [0x49f669] Error: BSONElement: bad type -128
Running db.system.profile.validate(
{full: true}) results in the following output:
{
"ns" : "songza.system.profile",
"capped" : true,
"max" : 2147483647,
"firstExtent" : "98:50708000 ns:songza.system.profile",
"lastExtent" : "98:50708000 ns:songza.system.profile",
"extentCount" : 1,
"extents" : [
{
"loc" : "98:50708000",
"xnext" : "null",
"xprev" : "null",
"nsdiag" : "songza.system.profile",
"size" : 4001792,
"firstRecord" : "98:507080b0",
"lastRecord" : "98:50872684"
}
],
"datasize" : 1454728,
"nrecords" : 1872,
"lastExtentSize" : 4001792,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "98:50708000",
"xnext" : "null",
"xprev" : "null",
"nsdiag" : "songza.system.profile",
"size" : 4001792,
"firstRecord" : "98:507080b0",
"lastRecord" : "98:50872684"
},
"cappedOutOfOrder" : 0,
"objectsFound" : 1872,
"invalidObjects" : 3,
"bytesWithHeaders" : 1484680,
"bytesWithoutHeaders" : 1454728,
"deletedCount" : 1,
"deletedSize" : 2516936,
"nIndexes" : 0,
"keysPerIndex" : {
},
"valid" : false,
"errors" : [
"invalid bson object detected (see logs for more info)"
],
"advice" : "ns corrupt, requires repair",
"ok" : 1
}
During the validation the following log lines are emitted:
Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id
I've also attached a copy of the affected collection to this jira.
- duplicates
-
-
- Closed
-