Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-8227

Assertion in mongo log: 10334:Invalid BSONObj size: 1769280243 (0xF3127569) first element: : ?type=100

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.2.2
    • Component/s: Querying
    • Environment:
      CentOS, x86_64,
    • Linux

      We have 6 servers in replica set, accessed using C++ and python drivers.
      One of servers sometimes throw in logs assertions like this:

      mongo.log

      Thu Jan 17 11:00:43 [conn103] Assertion: 10334:Invalid BSONObj size: 1769280243 (0xF3127569) first element: : ?type=100
      0xaffd31 0xac5eb9 0xac603c 0x572567 0x66f857 0x84c46b 0x84fcdd 0x85145f 0x851543 0x83dcf3 0x841fc1 0x844a27 0x7ade1f 0x7b4284 0x5703f2 0xaedfc1 0x355a607851 0x355a2e811d
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
      /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x99) [0xac5eb9]
      /usr/bin/mongod() [0xac603c]
      /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x497) [0x572567]
      /usr/bin/mongod(_ZNK5mongo11BSONElement14embeddedObjectEv+0xa7) [0x66f857]

      /usr/bin/mongod(_ZNK5mongo3Mod5applyERNS_15BSONBuilderBaseENS_11BSONElementERNS_8ModStateE+0x191b) [0x84c46b]

      /usr/bin/mongod(_ZN5mongo11ModSetState17createNewFromModsERKSsRNS_15BSONBuilderBaseERNS_18BSONIteratorSortedERKSt4pairIKSt17_Rb_tree_iteratorIS7_IS1_N5boost10shared_ptrINS_8ModStateEEEEESF_ERKNS_9LexNumCmpE+0x4dd)
      [0x84fcdd]

      /usr/bin/mongod(_ZN5mongo11ModSetState20createNewObjFromModsERKSsRNS_14BSONObjBuilderERKNS_7BSONObjE+0x7f) [0x85145f]
      /usr/bin/mongod(_ZN5mongo11ModSetState17createNewFromModsEv+0x63) [0x851543]
      /usr/bin/mongod() [0x83dcf3]

      /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x3081) [0x841fc1]

      /usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0x844a27]

      /usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x4cf) [0x7ade1f]

      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xbd4) [0x7b4284]

      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x5703f2]
      /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xaedfc1]
      /lib64/libpthread.so.0() [0x355a607851]
      /lib64/libc.so.6(clone+0x6d) [0x355a2e811d]
      Thu Jan 17 11:00:43 [conn103] Assertion: 10329:Element too large
      0xaffd31 0xac5eb9 0x5745c7 0x572fdb 0x65b17f 0x7b3aeb 0x5703f2 0xaedfc1 0x355a607851 0x355a2e811d
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
      /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x99) [0xac5eb9]

      /usr/bin/mongod(_ZNK5mongo7BSONObj8toStringERNS_17StringBuilderImplINS_16TrivialAllocatorEEEbbi+0x317) [0x5745c7]

      /usr/bin/mongod(_ZNK5mongo11BSONElement8toStringERNS_17StringBuilderImplINS_16TrivialAllocatorEEEbbi+0x69b) [0x572fdb]
      /usr/bin/mongod(_ZNK5mongo7OpDebug6reportERKNS_5CurOpE+0xaef) [0x65b17f]

      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x43b) [0x7b3aeb]

      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x5703f2]
      /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xaedfc1]
      /lib64/libpthread.so.0() [0x355a607851]
      /lib64/libc.so.6(clone+0x6d) [0x355a2e811d]

      We tracked down to piece of code which causes this, but it consists of 3 queries within single connections:

      schematic code

      conn->insert( "collTickets", ticketObj );
      conn->getLastError(); // ensure insertion was done
      conn->update( "userInfo", BSON( "_id" << userId ), BSON( "$inc" << BSON( "ticketCount" << 1 ) ), false/update mode/ );
      conn->update( "ticketAcks", BSON( "_id" << mongo::OID( ackIdStr ) ), BSON( "$set" << BSON( "lastTicket" << ticketOdj[ "_id" ].OID().str() << "accountant" << managerId ) ) );
      conn.done();

      objects sizes are guaranteed to be much less than 16MB:

      Mongo console
      truck:PRIMARY> Object.bsonsize( db.userInfo.findOne( { _id:1842412 } ) )
      43705
      truck:PRIMARY> db.collTickets.find( {userId:1842412} ).forEach( function ( e ) { print(Object.bsonsize(e));} )
      255
      356
      truck:PRIMARY> db.ticketAcks.find( {toUserId:1842412} ).forEach( function ( e ) { print(Object.bsonsize(e));} )
      269
      6732
      6178
      6414
      6334
      6100
      truck:PRIMARY> db.version()
      2.2.2
      
      

      What can cause this assertion failures?
      How we can avoid them?

            Assignee:
            Unassigned Unassigned
            Reporter:
            dink Dmitry Chernikov
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: