[SERVER-19049] Assertion: 10334:BSONObj size------- Mongodb state is "OTHER" Created: 19/Jun/15  Updated: 19/Jun/15  Resolved: 19/Jun/15

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: 2.6.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: xiaoli wang Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Conditionally Randomly Reoccure

How do I remove the valid data?

Participants:

 Description   

version: 2.6.8
3 nodes replica set
--------------- mongodb.log

2015-06-19T23:35:22.445+0800 [conn2313] ceilometer.meter 0xffec79 0xfa7ea5 0xf90716 0xf9084c 0x7dc8a7 0xdd8dfd 0xdd99d9 0xdc0fd9 0xdc1435 0xdc14ce 0xdc14ce 0xdc14ce 0xdc164f 0xb00d13 0xaf9014 0x8f756d 0x8dfa6a 0x8e037d 0x9e023e 0x9e0c09 
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xffec79]
 mongod(_ZN5mongo10logContextEPKc+0x1f5) [0xfa7ea5]
 mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0xf90716]
 mongod() [0xf9084c]
 mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x3a7) [0x7dc8a7]
 mongod(_ZNK5mongo5KeyV16toBsonEv+0xa3d) [0xdd8dfd]
 mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x39) [0xdd99d9]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findEPKNS_17IndexCatalogEntryERKNS_5KeyV1ERKNS_7DiskLocERib+0x109) [0xdc0fd9]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0x55) [0xdc1435]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0xee) [0xdc14ce]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0xee) [0xdc14ce]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0xee) [0xdc14ce]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_7BSONObjERiRbS8_i+0x7f) [0xdc164f]
 mongod(_ZNK5mongo18BtreeInterfaceImplINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_7BSONObjERiRbS8_i+0x73) [0xb00d13]
 mongod(_ZN5mongo22BtreeBasedAccessMethod5touchERKSt3setINS_7BSONObjENS_10BSONObjCmpESaIS2_EE+0xa4) [0xaf9014]
 mongod(_ZNK5mongo12IndexCatalog5touchEPKNS_16PregeneratedKeysE+0xed) [0x8f756d]
 mongod(_ZN5mongo10Collection15_insertDocumentERKNS_7BSONObjEbPKNS_16PregeneratedKeysE+0x4a) [0x8dfa6a]
 mongod(_ZN5mongo10Collection14insertDocumentERKNS_7BSONObjEbPKNS_16PregeneratedKeysE+0x1ad) [0x8e037d]
 mongod(_ZN5mongo18WriteBatchExecutor13execOneInsertEPNS0_16ExecInsertsStateEPPNS_16WriteErrorDetailE+0x21e) [0x9e023e]
 mongod(_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE+0x399) [0x9e0c09]
2015-06-19T23:35:22.445+0800 [conn2296] Assertion: 10334:BSONObj size: -2 (0xFFFFFFFE) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO
2015-06-19T23:35:22.445+0800 [conn2313] end connection 172.28.0.4:47945 (97 connections now open)
2015-06-19T23:35:22.455+0800 [conn2296] ceilometer.meter 0xffec79 0xfa7ea5 0xf90716 0xf9084c 0x7dc8a7 0xdd8dfd 0xdd99d9 0xdc0fd9 0xdc1435 0xdc14ce 0xdc14ce 0xdc14ce 0xdc164f 0xb00d13 0xaf9014 0x8f756d 0x8dfa6a 0x8e037d 0x9e023e 0x9e0c09 
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xffec79]
 mongod(_ZN5mongo10logContextEPKc+0x1f5) [0xfa7ea5]
 mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0xf90716]
 mongod() [0xf9084c]
 mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x3a7) [0x7dc8a7]
 mongod(_ZNK5mongo5KeyV16toBsonEv+0xa3d) [0xdd8dfd]
 mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x39) [0xdd99d9]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findEPKNS_17IndexCatalogEntryERKNS_5KeyV1ERKNS_7DiskLocERib+0x109) [0xdc0fd9]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0x55) [0xdc1435]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0xee) [0xdc14ce]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0xee) [0xdc14ce]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0xee) [0xdc14ce]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_7BSONObjERiRbS8_i+0x7f) [0xdc164f]
 mongod(_ZNK5mongo18BtreeInterfaceImplINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_7BSONObjERiRbS8_i+0x73) [0xb00d13]
 mongod(_ZN5mongo22BtreeBasedAccessMethod5touchERKSt3setINS_7BSONObjENS_10BSONObjCmpESaIS2_EE+0xa4) [0xaf9014]
 mongod(_ZNK5mongo12IndexCatalog5touchEPKNS_16PregeneratedKeysE+0xed) [0x8f756d]
 mongod(_ZN5mongo10Collection15_insertDocumentERKNS_7BSONObjEbPKNS_16PregeneratedKeysE+0x4a) [0x8dfa6a]
 mongod(_ZN5mongo10Collection14insertDocumentERKNS_7BSONObjEbPKNS_16PregeneratedKeysE+0x1ad) [0x8e037d]
 mongod(_ZN5mongo18WriteBatchExecutor13execOneInsertEPNS0_16ExecInsertsStateEPPNS_16WriteErrorDetailE+0x21e) [0x9e023e]
 mongod(_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE+0x399) [0x9e0c09]
2015-06-19T23:35:22.455+0800 [conn2296] insert ceilometer.meter query: { _id: ObjectId('558436ba97f6435eb47a0ba5'), counter_name: "quota.ram", user_id: null, message_signature: "f92ed97306f2b348b577236d49aaf682e539bfce361b2be0024ec670146d1d99", timestamp: new Date(1434725710000), resource_id: "398408b47f1e4bc4a5b50c0447f1e3d1", resource_metadata: { limitCores: 20480, limitRam: 25165824, totalCoresUsed: 0, totalRAMUsed: 0, limitInstances: 2048, totalInstancesUsed: 0, domain_id: "default", additionalInfo: "" }, source: "openstack", counter_unit: "%", counter_volume: 0.0, recorded_at: new Date(1434728122346), project_id: "398408b47f1e4bc4a5b50c0447f1e3d1", message_id: "2eaceaa4-1693-11e5-867f-9017acae562f", counter_type: "gauge" } ninserted:0 keyUpdates:0 exception: BSONObj size: -2 (0xFFFFFFFE) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO code:10334 numYields:0 locks(micros) w:9864 108ms
2015-06-19T23:35:22.455+0800 [conn2301] Assertion: 10334:BSONObj size: -2 (0xFFFFFFFE) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO
2015-06-19T23:35:22.455+0800 [conn2296] command ceilometer.$cmd command: insert { insert: "meter", ordered: true, documents: [ { _id: ObjectId('558436ba97f6435eb47a0ba5'), counter_name: "quota.ram", user_id: null, message_signature: "f92ed97306f2b348b577236d49aaf682e539bfce361b2be0024ec670146d1d99", timestamp: new Date(1434725710000), resource_id: "398408b47f1e4bc4a5b50c0447f1e3d1", resource_metadata: { limitCores: 20480, limitRam: 25165824, totalCoresUsed: 0, totalRAMUsed: 0, limitInstances: 2048, totalInstancesUsed: 0, domain_id: "default", additionalInfo: "" }, source: "openstack", counter_unit: "%", counter_volume: 0.0, recorded_at: new Date(1434728122346), project_id: "398408b47f1e4bc4a5b50c0447f1e3d1", message_id: "2eaceaa4-1693-11e5-867f-9017acae562f", counter_type: "gauge" } ] } keyUpdates:0 numYields:0  reslen:242 108ms
2015-06-19T23:35:22.465+0800 [conn2296] end connection 172.28.0.4:47922 (96 connections now open)
2015-06-19T23:35:22.465+0800 [conn2301] ceilometer.meter 0xffec79 0xfa7ea5 0xf90716 0xf9084c 0x7dc8a7 0xdd8dfd 0xdd99d9 0xdc0fd9 0xdc1435 0xdc14ce 0xdc14ce 0xdc14ce 0xdc164f 0xb00d13 0xaf9014 0x8f756d 0x8dfa6a 0x8e037d 0x9e023e 0x9e0c09 
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xffec79]
 mongod(_ZN5mongo10logContextEPKc+0x1f5) [0xfa7ea5]
 mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0xf90716]
 mongod() [0xf9084c]
 mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x3a7) [0x7dc8a7]
 mongod(_ZNK5mongo5KeyV16toBsonEv+0xa3d) [0xdd8dfd]
 mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x39) [0xdd99d9]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findEPKNS_17IndexCatalogEntryERKNS_5KeyV1ERKNS_7DiskLocERib+0x109) [0xdc0fd9]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0x55) [0xdc1435]
 mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateEPKNS_17IndexCatalogEntryERKNS_7DiskLocERKNS_5KeyV1ERiRbS8_i+0xee) [0xdc14ce]



 Comments   
Comment by Ramon Fernandez Marina [ 19/Jun/15 ]

esala116, this message indicates that there's data corruption in one of the indices for the ceilometer.meter namespace; this could have been caused by flaky storage/memory. You can rebuild the indexes on this namespace either directly, or if this may have a severe impact in your primary by using this other procedure.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Generated at Thu Feb 08 03:49:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.