[SERVER-5452] Assertion: 10334:Invalid BSONObj error when doing Map/Reduce Created: 29/Mar/12  Updated: 10/Aug/12  Resolved: 30/Apr/12

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: David Riley Assignee: Antoine Girbal
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RHEL 5.8


Issue Links:
Related
related to SERVER-6750 Clarify message for error 10334 - Inv... Closed
Operating System: Linux
Participants:

 Description   

Hello all,

I have been using Mongo in non-sharded mode for quite sometime and have just recently tried my hand at sharding data. To do so I have 3 machines setup as mongod with 1 of them also serving as my mongos and config server.

I have ~212 thousand documents in the sharded collection but each document has what can be a pretty long array.

This dataset was working fine for about a day but after doing an update where I added a new field to most of the documents I am getting the following error:

Thu Mar 29 14:56:22 [conn5166] Assertion: 10334:Invalid BSONObj size: 18303487 (0xFF491701) first element: 0:

{ count: 6294467.0 }

The complete log from mongos is here:

Thu Mar 29 14:56:22 [conn5166] Assertion: 10334:Invalid BSONObj size: 18303487 (0xFF491701) first element: 0: { count: 6294467.0 }
0x584432 0x508461 0x9bacbd 0x9bb0c8 0x9b3e87 0x9b43a5 0x9ca9bb 0x97cc14 0x97e20f 0x940e25 0x9441b1 0x8869d7 0x88df49 0xaa37d6 0x637497 0x3d32e0677d 0x3d31ed49ad 
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x584432]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x471) [0x508461]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo2mr9JSReducer7_reduceERKSt6vectorINS_7BSONObjESaIS3_EERS3_Ri+0xf2d) [0x9bacbd]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo2mr9JSReducer6reduceERKSt6vectorINS_7BSONObjESaIS3_EE+0x88) [0x9bb0c8]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo2mr5State14reduceInMemoryEv+0xe7) [0x9b3e87]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo2mr5State9checkSizeEv+0xf5) [0x9b43a5]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xa1b) [0x9ca9bb]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x6a4) [0x97cc14]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97e20f]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x940e25]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x11e1) [0x9441b1]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod [0x8869d7]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88df49]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa37d6]
 /export/mongodb-linux-x86_64-2.0.3/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x637497]
 /lib64/libpthread.so.0 [0x3d32e0677d]
 /lib64/libc.so.6(clone+0x6d) [0x3d31ed49ad]
Thu Mar 29 14:56:22 [conn5166] warning: ClientCursor::YieldLock not closed properly
Thu Mar 29 14:56:22 [conn5166] mr failed, removing collection :: caused by :: 10334 Invalid BSONObj size: 18303487 (0xFF491701) first element: 0: { count: 6294467.0 }
Thu Mar 29 14:56:22 [conn5166] CMD: drop lgt8.tmp.mr.bwa_mapping_49
Thu Mar 29 14:56:22 [conn5166] CMD: drop lgt8.tmp.mr.bwa_mapping_49_inc
Thu Mar 29 14:56:22 [conn5166] command lgt8.$cmd command: { mapreduce: "bwa_mapping", map: "    function() {
        var conds = [];
        var thiselm = this;
        this.hits.forEach(
            function(h) {
                var good = t...", reduce: "    function(key,values) {
        var result = {count:0.0};
        values.forEach(function(value) {
            result.count += value.count;
       ...", query: {}, out: "tmp.mrs.bwa_mapping_1333047154_116" } ntoreturn:1 exception: Invalid BSONObj size: 18303487 (0xFF491701) first element: 0: { count: 6294467.0 } code:10334 reslen:187 228442ms

I have tried running repairDatabase(), which ran successfully but does not take care of the problem. I have also run validate() on the collection and validate() reports everything as OK (even when run in verbose mode). I am able to query this database without error however I am unable to perform the map reduce described above (which is counting a subset of the array elements and doing a group by). I am able to perform a map reduce on the elements of the document NOT contained in the array.

I am not sure how much of this information is helpful but I thought I would include it just in case.

Any help on this would be greatly appreciated.



 Comments   
Comment by Nic Cottrell (Personal) [ 09/Aug/12 ]

I just got this error too. Would it be possible to change the error to something like "BSON size exceeds max of 16MB" ?

Comment by Antoine Girbal [ 30/Apr/12 ]

this error means that one document being created during MR is larger than the max size of 16MB.
You should try to break any large document up in smaller ones with the emit key, or change the process in some way.

Comment by David Riley [ 30/Mar/12 ]

Hi Eliot,

The map and reduce functions are below, note that the look a little funky because they are embedded in a perl script:

Map:

    function() {
        var conds = [];
        var thiselm = this;
        this.hits.forEach(
            function(h) {
                var good = true;
                if(conds.length > 0) {
                    good = true;
                    conds.forEach(
                        function(c) {
                            if(h[c.key] != c.value) {                            
                                good = false;
                            }
                    });
                }
                if(good) {
                    emit(h.genus, {count:1});
                }
            }
        );
    }

Reduce:

    function(key,values) {
        var result = {count:0.0};
        values.forEach(function(value) {
            result.count += value.count;
                       });
        return result;
    }

So the idea in that map is that I can pass in some conditions (as key value pairs) to check for in the array elements. I a a particular value from the array element and that is what is getting counted.

Comment by Eliot Horowitz (Inactive) [ 30/Mar/12 ]

Can you send the map and reduce function you are using?

Generated at Thu Feb 08 03:08:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.