MR parallel processing failed

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • 2.5.4
    • Affects Version/s: None
    • Component/s: MapReduce
    • None
    • Environment:
      Linux precise64 3.2.0-23-generic #36-Ubuntu
    • ALL
    • Hide

      apt-get install mongodb-10gen
      mkdir -p /data/

      {configdb,datadb1,datadb2}

      mongod --port 27019 --configsvr > configdb.out &
      mongod --port 27017 --dbpath /data/datadb1 > datadb1.out &
      mongod --port 27016 --dbpath /data/datadb2 > datadb2.out &
      mongos --port 27018 --configdb 127.0.0.1:27019 > mongos.out &

      mongo --port 27018

      sh.addShard('127.0.0.1:27017');
sh.addShard('127.0.0.1:27016');
use test;
db.test.ensureIndex({'geo': 1});
db.test.ensureIndex({ geo: "2d", time: 1 });
db.test.ensureIndex({'time': 1});
sh.enableSharding('test');
sh.shardCollection('test.test', {'_id': 1});
for (var i=0; i<1000000; i++) {
	db.test.insert({time: Math.round(i/1000), geo: [Math.random()*90, Math.random()*90]});
}
db.runCommand({
	mapreduce: 'test',
	map: function() { emit(this.time, {count: 1}) },
	reduce: function(key, values) {
		var count = 0;
     	values.forEach(function(v) {
        	count += v['count'];
     	});
     	return {count: count};
	},
	query: { "geo": { "$within": { "$box": [ [-90,-90] , [90,90]] } } },
	out: 'inline'
});
{
	"ok" : 0,
	"errmsg" : "MR parallel processing failed: { errmsg: \"exception: can't get a DBWrite while having a read lock\", code: 16186, ok: 0.0 }"
}
      Show
      apt-get install mongodb-10gen mkdir -p /data/ {configdb,datadb1,datadb2} mongod --port 27019 --configsvr > configdb.out & mongod --port 27017 --dbpath /data/datadb1 > datadb1.out & mongod --port 27016 --dbpath /data/datadb2 > datadb2.out & mongos --port 27018 --configdb 127.0.0.1:27019 > mongos.out & mongo --port 27018 sh.addShard( '127.0.0.1:27017' ); sh.addShard( '127.0.0.1:27016' ); use test; db.test.ensureIndex({ 'geo' : 1}); db.test.ensureIndex({ geo: "2d" , time: 1 }); db.test.ensureIndex({ 'time' : 1}); sh.enableSharding( 'test' ); sh.shardCollection( 'test.test' , { '_id' : 1}); for ( var i=0; i<1000000; i++) { db.test.insert({time: Math .round(i/1000), geo: [ Math .random()*90, Math .random()*90]}); } db.runCommand({ mapreduce: 'test' , map: function() { emit( this .time, {count: 1}) }, reduce: function(key, values) { var count = 0; values.forEach(function(v) { count += v[ 'count' ]; }); return {count: count}; }, query: { "geo" : { "$within" : { "$box" : [ [-90,-90] , [90,90]] } } }, out: 'inline' }); { "ok" : 0, "errmsg" : "MR parallel processing failed: { errmsg: \" exception: can't get a DBWrite while having a read lock\ ", code: 16186, ok: 0.0 }" }
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When specifying a query for the map-reduce job, it fails. Without query it works flawlessly, unfortunately without the query it runs miserably slow on our dataset.

      I am simply aggregating the records based on one of the fields, and filter them with the query based on a 2d index. It fails. Now I'm working with a workaround to filter the records in the map() function, which obviously sucks.

      I have brought to you a full step-by-step reproduction guide. It seems there is no such problem on a single instance of mongodb, as well as with a smaller dataset.

            Assignee:
            David Storch
            Reporter:
            Attila Incze
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: