[SERVER-7913] Invalid BSON in db.system.profile Created: 12/Dec/12  Updated: 10/Dec/14  Resolved: 15/Mar/13

Status: Closed
Project: Core Server
Component/s: Logging
Affects Version/s: 2.2.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Michael Henson Assignee: Tad Marshall
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux - Ubuntu Lucid
Linux db1e.songza.com 2.6.32-348-ec2 #54-Ubuntu SMP Fri Sep 7 11:21:33 UTC 2012 x86_64 GNU/Linux


Attachments: File system.profile.bson.bz2    
Issue Links:
Duplicate
duplicates SERVER-7206 potential mongod crash in profiling, 2.2 Closed
Operating System: ALL
Participants:

 Description   

We are getting what appear to be invalid bson documents in the system profile collection, which is making it rather difficult to analyze the results.

After enabling the profiler, these invalid documents will appear after a few minutes of run time. Dropping the capped collection and recreating it seems to have no effect as the invalid documents will be reinserted after a few minutes time. So far we have only seen this issue on our primary node. The secondary machines in our replica set do not appear to be affected. I don't recall ever getting this sort of error when we were running mongo 2.0. This is the first time since our upgrade to 2.2.1 that we have turned on the profiler.

Attempts to iterate through db.system.profile in the mongo shell result in the following assertion:

Wed Dec 12 16:16:59 Assertion: 10320:BSONElement: bad type -128
0x6073f1 0x5d1aa9 0x4b0d98 0x5c17a6 0x6b3f35 0x6b6a2c 0x69be0a 0x6aab74 0x695ebe 0x6aa5ac 0x69c025 0x6aa13f 0x668e46 0x668ec2 0x66a2ce 0x5cbcc4 0x4a47a5 0x4a67e6 0x7feb2ccf9c4d 0x49f669 
 mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x6073f1]
 mongo(_ZN5mongo11msgassertedEiPKc+0x99) [0x5d1aa9]
 mongo(_ZNK5mongo11BSONElement4sizeEv+0x1d8) [0x4b0d98]
 mongo(_ZN5mongo16resolveBSONFieldEP9JSContextP8JSObjectljPS3_+0x146) [0x5c17a6]
 mongo(js_LookupPropertyWithFlags+0x3f5) [0x6b3f35]
 mongo(js_GetProperty+0x7c) [0x6b6a2c]
 mongo(js_Interpret+0x10ea) [0x69be0a]
 mongo(js_Invoke+0x9d4) [0x6aab74]
 mongo() [0x695ebe]
 mongo(js_Invoke+0x40c) [0x6aa5ac]
 mongo(js_Interpret+0x1305) [0x69c025]
 mongo(js_Execute+0x36f) [0x6aa13f]
 mongo(JS_EvaluateUCScriptForPrincipals+0x66) [0x668e46]
 mongo(JS_EvaluateUCScript+0x22) [0x668ec2]
 mongo(JS_EvaluateScript+0x6e) [0x66a2ce]
 mongo(_ZN5mongo7SMScope4execERKNS_10StringDataERKSsbbbi+0x144) [0x5cbcc4]
 mongo(_Z5_mainiPPc+0x2455) [0x4a47a5]
 mongo(main+0x26) [0x4a67e6]
 /lib/libc.so.6(__libc_start_main+0xfd) [0x7feb2ccf9c4d]
 mongo(__gxx_personality_v0+0x2a1) [0x49f669]
Error: BSONElement: bad type -128

Running db.system.profile.validate(

{full: true}

) results in the following output:

{
	"ns" : "songza.system.profile",
	"capped" : true,
	"max" : 2147483647,
	"firstExtent" : "98:50708000 ns:songza.system.profile",
	"lastExtent" : "98:50708000 ns:songza.system.profile",
	"extentCount" : 1,
	"extents" : [
		{
			"loc" : "98:50708000",
			"xnext" : "null",
			"xprev" : "null",
			"nsdiag" : "songza.system.profile",
			"size" : 4001792,
			"firstRecord" : "98:507080b0",
			"lastRecord" : "98:50872684"
		}
	],
	"datasize" : 1454728,
	"nrecords" : 1872,
	"lastExtentSize" : 4001792,
	"padding" : 1,
	"firstExtentDetails" : {
		"loc" : "98:50708000",
		"xnext" : "null",
		"xprev" : "null",
		"nsdiag" : "songza.system.profile",
		"size" : 4001792,
		"firstRecord" : "98:507080b0",
		"lastRecord" : "98:50872684"
	},
	"cappedOutOfOrder" : 0,
	"objectsFound" : 1872,
	"invalidObjects" : 3,
	"bytesWithHeaders" : 1484680,
	"bytesWithoutHeaders" : 1454728,
	"deletedCount" : 1,
	"deletedSize" : 2516936,
	"nIndexes" : 0,
	"keysPerIndex" : {
		
	},
	"valid" : false,
	"errors" : [
		"invalid bson object detected (see logs for more info)"
	],
	"advice" : "ns corrupt, requires repair",
	"ok" : 1
}

During the validation the following log lines are emitted:

Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id
Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id
Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id

I've also attached a copy of the affected collection to this jira.



 Comments   
Comment by Tad Marshall [ 15/Mar/13 ]

Hi Michael,

Thanks for letting us know!

Tad

Comment by Michael Henson [ 14/Mar/13 ]

Following up here ... We upgraded to 2.2.2 and this issue looks to be resolved. You can close this one out.

Comment by Michael Henson [ 12/Dec/12 ]

Sure. We'll start the upgrade process within the next week. I'll follow back up after that process has been completed.

Comment by Tad Marshall [ 12/Dec/12 ]

Hi Michael,

This sounds a bit like SERVER-7206, although you are not seeing the crash at this point.

Can you try version 2.2.2 and let us know if the problem goes away?

Thanks!

Tad

Generated at Thu Feb 08 03:15:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.