Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10980

MR parallel processing failed

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • 2.5.4
    • None
    • MapReduce
    • None
    • Linux precise64 3.2.0-23-generic #36-Ubuntu
    • ALL
    • Hide

      apt-get install mongodb-10gen
      mkdir -p /data/

      {configdb,datadb1,datadb2}

      mongod --port 27019 --configsvr > configdb.out &
      mongod --port 27017 --dbpath /data/datadb1 > datadb1.out &
      mongod --port 27016 --dbpath /data/datadb2 > datadb2.out &
      mongos --port 27018 --configdb 127.0.0.1:27019 > mongos.out &

      mongo --port 27018

      sh.addShard('127.0.0.1:27017');
      sh.addShard('127.0.0.1:27016');
      use test;
      db.test.ensureIndex({'geo': 1});
      db.test.ensureIndex({ geo: "2d", time: 1 });
      db.test.ensureIndex({'time': 1});
      sh.enableSharding('test');
      sh.shardCollection('test.test', {'_id': 1});
      for (var i=0; i<1000000; i++) {
      	db.test.insert({time: Math.round(i/1000), geo: [Math.random()*90, Math.random()*90]});
      }
      db.runCommand({
      	mapreduce: 'test',
      	map: function() { emit(this.time, {count: 1}) },
      	reduce: function(key, values) {
      		var count = 0;
           	values.forEach(function(v) {
              	count += v['count'];
           	});
           	return {count: count};
      	},
      	query: { "geo": { "$within": { "$box": [ [-90,-90] , [90,90]] } } },
      	out: 'inline'
      });
      {
      	"ok" : 0,
      	"errmsg" : "MR parallel processing failed: { errmsg: \"exception: can't get a DBWrite while having a read lock\", code: 16186, ok: 0.0 }"
      }

      Show
      apt-get install mongodb-10gen mkdir -p /data/ {configdb,datadb1,datadb2} mongod --port 27019 --configsvr > configdb.out & mongod --port 27017 --dbpath /data/datadb1 > datadb1.out & mongod --port 27016 --dbpath /data/datadb2 > datadb2.out & mongos --port 27018 --configdb 127.0.0.1:27019 > mongos.out & mongo --port 27018 sh.addShard('127.0.0.1:27017'); sh.addShard('127.0.0.1:27016'); use test; db.test.ensureIndex({'geo': 1}); db.test.ensureIndex({ geo: "2d", time: 1 }); db.test.ensureIndex({'time': 1}); sh.enableSharding('test'); sh.shardCollection('test.test', {'_id': 1}); for (var i=0; i<1000000; i++) { db.test.insert({time: Math.round(i/1000), geo: [Math.random()*90, Math.random()*90]}); } db.runCommand({ mapreduce: 'test', map: function() { emit(this.time, {count: 1}) }, reduce: function(key, values) { var count = 0; values.forEach(function(v) { count += v['count']; }); return {count: count}; }, query: { "geo": { "$within": { "$box": [ [-90,-90] , [90,90]] } } }, out: 'inline' }); { "ok" : 0, "errmsg" : "MR parallel processing failed: { errmsg: \"exception: can't get a DBWrite while having a read lock\", code: 16186, ok: 0.0 }" }

    Description

      When specifying a query for the map-reduce job, it fails. Without query it works flawlessly, unfortunately without the query it runs miserably slow on our dataset.

      I am simply aggregating the records based on one of the fields, and filter them with the query based on a 2d index. It fails. Now I'm working with a workaround to filter the records in the map() function, which obviously sucks.

      I have brought to you a full step-by-step reproduction guide. It seems there is no such problem on a single instance of mongodb, as well as with a smaller dataset.

      Attachments

        Activity

          People

            david.storch@mongodb.com David Storch
            atimb Attila Incze
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: