[SERVER-12615] Hitting Max Document size upon Map/Reduce? Created: 04/Feb/14  Updated: 11/Jul/16  Resolved: 19/Mar/14

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: 2.5.5
Fix Version/s: 2.6.0-rc2

Type: Bug Priority: Major - P3
Reporter: Brad C. Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.5 x86_64 Sharded


Issue Links:
Duplicate
duplicates SERVER-12949 MapReduce not doing incremental reduc... Closed
Backwards Compatibility: Fully Compatible
Participants:

 Description   

I'm running an incremental map reduce over a large data set in a sharded environment. The output is set to "reduce".

This strategy works fine until the batch I'm running the MR on exceeds ~14Million documents. At which point the MR will fail with error code 10334 saying:

"MR Parallel Processing failed. errmsg: 'Exception: BSONObj size 16951756 ... is invalid ..."

I was under the impression that there are no size concerns when it comes to map/reduce. (Assuming of course your document size doesn't exceed 16MB. All of the documents I'm dealing with are ~400bytes).

It doesn't fail immediately and I suspect it's the cumulative data from one shard or another that is exceeding this size. I wasn't aware that this was an issue with MR in sharded environments.

Any ideas what's going on?



 Comments   
Comment by Githook User [ 19/Mar/14 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-12615 Fix size calculation in JSReducer::_reduce

The old calculation didn't consider the size of the numeric-string fieldnames
in arrays. This error accumulated with each value added to the array. While
the new calculation doesn't consider the size of the current fieldname, it is
ok because it prevents the error from accumulating and we have plenty of room
between the User and Internal max sizes.

(cherry picked from commit 3e0aa5769dd58c727726c3397f9678d7f767b7ff)
Branch: v2.6
https://github.com/mongodb/mongo/commit/0147b057c2d01fc6cd418af09fbee278a877baf1

Comment by Githook User [ 19/Mar/14 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-12615 Fix size calculation in JSReducer::_reduce

The old calculation didn't consider the size of the numeric-string fieldnames
in arrays. This error accumulated with each value added to the array. While
the new calculation doesn't consider the size of the current fieldname, it is
ok because it prevents the error from accumulating and we have plenty of room
between the User and Internal max sizes.
Branch: master
https://github.com/mongodb/mongo/commit/3e0aa5769dd58c727726c3397f9678d7f767b7ff

Comment by Mathias Stearn [ 14/Mar/14 ]

Turns out that while SERVER-12949 caused this to present and fixing it makes this error go away, there is a separate bug in how we compute sizes in JSReducer::_reduce().

Comment by Mathias Stearn [ 14/Mar/14 ]

Please try again with 2.6.0-rc2 once it is released. If this still occurs, please reopen.

Comment by Mathias Stearn [ 14/Mar/14 ]

I think this is a side-effect of SERVER-12949 which should be fixed soon.

Comment by Brad C. [ 05/Feb/14 ]

Some additional info.

  • The error seems to get thrown during the "2/3 final reduce in memory" step.
  • It happens regardless of weather the "reduce" collection is empty or has documents in it already.
Comment by Brad C. [ 05/Feb/14 ]

Thanks for the reply and apologies that this info wasn't provided initially. It's somewhat sensitive in nature so here's a scrubbed version which should be consistent with what we're running.

var db = db.getSiblingDB("sandbox");
 
var map = function() {
    var data = this;
    if(!this._lid) this._lid = null;
    if(!this.a) this.a = null;
    if(!this.b) this.b = null;
    if(!this.c) this.c = null;
    if(!this.d) this.d = null;
 
    emit( {'a':this.a, 'b':this.b, 'c':this.c, 'd':this.d}, 
          {'data':data, '_lid':this._lid }
    );
}
 
var reduce = function (idkeys, attrs){
  var newattrs = null;
  attrs.forEach(function(doc) {
      if(!newattrs) {
        newattrs = doc;
      }
      else {
        var orig_lid = newattrs._lid;
        var new_lid = doc._lid;
        if((doc.data.t > newattrs.data.t) || (doc.data.t.getTime() === newattrs.data.t.getTime())) { 
          newattrs = doc;
          
          if(orig_lid) {
            newattrs._lid = orig_lid;
          }
        }
      }
  });
  return newattrs;
}
 
var min = ObjectId(....);
var max = ObjectId(....);
 
var ret = db.testdb.mapReduce(  
            map,
            reduce,
            {
              query: {_id:{$gt: min, $lte: max }, 'Position':{$exists:1}},
              out: { reduce: 'agg_testdb', nonAtomic:true},
              //sort: { a:1, b:1, c:1, d:1 },
            }
)

For clarification, the line:

var data = this;

Is there to capture all fields in the original document without knowing what it might contain beforehand. Since this seemed to be the most likely culprit for some sort of recursion, I tried omitting this particular line and just explicitly emitting attributes from the document but the same error occurs.

If it were recursing in some fashion I wouldn't expect it to work for 10 million documents and produce the expected output but not work for 15 million.

It may also be worth noting a bug/peculiarity I've observed when viewing the status of the MR in db.currentOP() where "Emit progress" climbs into the millions of % complete.

		{
			"opid" : "shard0009:4308900",
			"active" : true,
			"secs_running" : 20,
			"op" : "query",
			"ns" : "sandbox.testdb",
			"query" : {
				"$msg" : "query not recording (too large)"
			},
			"client_s" : "192.168.24.101:39500",
			"desc" : "conn32",
			"threadId" : "0x7fb14807e700",
			"connectionId" : 32,
			"locks" : {
				"^" : "r",
				"^npr" : "R"
			},
			"waitingForLock" : false,
			"msg" : "m/r: (1/3) emit phase M/R: (1/3) Emit Progress: 123636/1 12363600%",
			"progress" : {
				"done" : 123636,
				"total" : 1
			},
			"numYields" : 204,
			"lockStats" : {
				"timeLockedMicros" : {
					"r" : NumberLong(40113420),
					"w" : NumberLong(2006)
				},
				"timeAcquiringMicros" : {
					"r" : NumberLong(101675),
					"w" : NumberLong(5)
				}
			}
		},
		{
			"opid" : "shard0010:2830042",
			"active" : true,
			"secs_running" : 20,
			"op" : "query",
			"ns" : "sandbox.testdb",
			"query" : {
				"$msg" : "query not recording (too large)"
			},
			"client_s" : "127.0.0.1:39159",
			"desc" : "conn1",
			"threadId" : "0x7feb5fc6c700",
			"connectionId" : 1,
			"locks" : {
				"^" : "r",
				"^npr" : "R"
			},
			"waitingForLock" : false,
			"msg" : "m/r: (1/3) emit phase M/R: (1/3) Emit Progress: 131242/1 13124200%",
			"progress" : {
				"done" : 131242,
				"total" : 1
			},
			"numYields" : 260,
			"lockStats" : {
				"timeLockedMicros" : {
					"r" : NumberLong(40131796),
					"w" : NumberLong(2039)
				},
				"timeAcquiringMicros" : {
					"r" : NumberLong(84119),
					"w" : NumberLong(5)
				}
			}
		},

It's suggested in your MR documentation that pre-sorting the data by the emit key may be beneficial. As you can see, that line is commented out because when it's active it's incredibly slow even though there is a compound index on a:1,b:1,c:1,d:1.

Comment by Mathias Stearn [ 04/Feb/14 ]

Could you include a set of steps to reproduce, including the exact command you are sending to trigger the mapReduce? Sometimes these errors can be caused by errors in the map or reduce functions that cause the output to be much larger than expected.

Generated at Thu Feb 08 03:29:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.