[SERVER-7548] Assertion: 10334:Invalid BSONObj size: -286331391 (0x01EEEEEE) Created: 04/Nov/12  Updated: 15/Feb/13  Resolved: 14/Nov/12

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andy Gray Assignee: Tad Marshall
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.3


Issue Links:
Duplicate
duplicates SERVER-7206 potential mongod crash in profiling, 2.2 Closed
Operating System: ALL
Participants:

 Description   

We have a replica set running on 5 hosts. The current primary is running 2.0.4 (soon to be upgraded) and the secondaries are running 2.2.0. Over the weekend one of the secondaries started throwing this exception for all queries:

Mon Nov  5 09:06:35 [conn193437] Assertion: 10334:Invalid BSONObj size: -286331391 (0x01EEEEEE) first element: : ?type=-108
0xade6e1 0x8036eb 0x80386c 0x571bc7 0xadf73d 0x7474e4 0xa61af4 0x62dc57 0x62dc98 0x73954c 0x70f40f 0xb5db70 0x56fa52 0x5dbf11 0x3cb0407851 0x3cb00e811d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0x8036eb]
 /usr/bin/mongod() [0x80386c]
 /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x497) [0x571bc7]
 /usr/bin/mongod(_ZN5mongo13unindexRecordEPNS_16NamespaceDetailsEPNS_6RecordERKNS_7DiskLocEb+0x11d) [0xadf73d]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x204) [0x7474e4]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0xa61af4]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /usr/bin/mongod(_ZN5mongo7profileERKNS_6ClientERNS_5CurOpE+0x95f) [0x70f40f]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xec0) [0xb5db70]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fa52]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0x5dbf11]
 /lib64/libpthread.so.0() [0x3cb0407851]
 /lib64/libc.so.6(clone+0x6d) [0x3cb00e811d]
Mon Nov  5 09:06:35 [conn193437] AssertionException handling request, closing client connection: 10334 Invalid BSONObj size: -286331391 (0x01EEEEEE) first element: : ?type=-108

Restarting the mongod process fixed the issue. There appears to be older issues relating to this exception, but they aren't for 2.2.0

I've looked back in the logs to the first occurrence of this and there doesn't seem to be anything unusual happening prior to the error.

Is this a known issue? It appeared to have no direct cause which is a little bit concerning. Are there any suggestions for how to avoid this happening again?



 Comments   
Comment by Andy Gray [ 14/Nov/12 ]

Excellent. Thanks for the thorough answer!

Comment by Tad Marshall [ 14/Nov/12 ]

This is caused by corruption in the profile collection, fixed in SERVER-7206. To avoid seeing this error in the future, you need to drop the profile collection and upgrade to the upcoming version 2.2.2 or newer before using profiling again. A recent nightly build in the 2.2 series will include the fix, and release candidate 0 of 2.2.2 is imminent.

Comment by Tad Marshall [ 05/Nov/12 ]

Seems to be the same profiling crash as SERVER-7206.

Generated at Thu Feb 08 03:14:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.