[SERVER-16429] Concurrent mapReduces with same replacement collection fail with WT Created: 05/Dec/14  Updated: 18/Dec/15  Resolved: 10/Dec/14

Status: Closed
Project: Core Server
Component/s: Concurrency, MapReduce
Affects Version/s: 2.8.0-rc2
Fix Version/s: 2.8.0-rc3

Type: Bug Priority: Major - P3
Reporter: Kamran K. Assignee: Kaloian Manassiev
Resolution: Done Votes: 0
Labels: 28qa, wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-16507 initial sync needs to lock properly b... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Run this script against mongod --storageEngine=wiredTiger:

'use strict';
 
load('jstests/libs/parallelTester.js');
 
function client() {
 
    function mapper() {
        emit(this.key, 1);
    }
 
    function reducer() {
        return {};
    }
 
    while (true) {
        var options = {
            out: {replace: 'bar'}
        };
 
        db.foo.mapReduce(mapper, reducer, options);
    }
}
 
db.foo.drop();
 
var bulk = db.foo.initializeUnorderedBulkOp();
for (var i = 0; i < 3000; i++) {
    bulk.insert({i: i, key: Random.randInt()});
}
bulk.execute();
 
var threads = [];
for (var i = 0; i < 7; i++) {
    var t = new ScopedThread(client);
    threads.push(t);
    t.start();
}
 
threads.forEach(function(t) {
    t.join();
});

Participants:

 Description   

Concurrent mapReduce operations that specify the same replacement collection can fail with "target namespace exists" errors when using WiredTiger. The same error is seemingly not reproducible with mmapv1.

I wouldn't expect a mapReduce operation with a 'replace' action to fail due to an existing namespace.

Error message:

Error: map reduce failed:{
	"errmsg" : "exception: rename failed: { ok: 0.0, errmsg: \"target namespace exists\" }",
	"code" : 10076,
	"ok" : 0
}


Version: b766e395167dc27



 Comments   
Comment by Moshe Kaplan [X] [ 18/Dec/15 ]

Dan Pasette, just did it: https://jira.mongodb.org/browse/SERVER-21943
10x

Comment by Daniel Pasette (Inactive) [ 17/Dec/15 ]

MosheKaplan, This doesn't look like precisely the same issue. Can you open a new SERVER ticket where we can get additional details from you?

Comment by Moshe Kaplan [X] [ 17/Dec/15 ]

We got the same error at version 3.0.4.
It happens once a week in a sharded environment with WT engine in the mongod instances.
The evidences is a crashed shard, while in the other shard there a remaining of a tmp table that was not deleted.

Attached is the log error from the failed shard:

2015-12-12T00:35:40.814+0000 I COMMAND  [conn92487] mr failed, removing collection :: caused by :: WriteConflict
2015-12-12T00:35:40.818+0000 I COMMAND  [conn92487] CMD: drop XXXX.tmp.mr.account_231015
2015-12-12T00:35:40.822+0000 I NETWORK  [initandlisten] connection accepted from XXX.XXX.XXX.XXX:XXXXX #110728 (47 connections now open)
2015-12-12T00:35:40.920+0000 I COMMAND  [conn92487] command XXXX.$cmd command: drop { drop: "tmp.mr.account_231015" } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:122 locks:{ Global: { acquireCount: { r: 8, w: 4 } }, Database: { acquireCount: { r: 1, w: 1, R: 1, W: 4 }, acquireWaitCount: { W: 4 }, timeAcquiringMicros: { W: 7627049380 } }, Collection: { acquireCount: { r: 1, w: 1, W: 1 } } } 102ms
2015-12-12T00:35:40.920+0000 I QUERY    [conn110694] query XXXX.endpoints query: { $query: { gw: { $gt: 0 }, $or: [ { status: "unmanaged" }, { status: "managed" } ] }, $readPreference: { mode: "secondaryPreferred" } } planSummary: IXSCAN { gw: -1.0, status: -1.0 } ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:0 keyUpdates:0 writeConflicts:0 numYields:1 nreturned:0 reslen:20 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 }, acquireWaitCount: { r: 2 }, timeAcquiringMicros: { r: 3690878996 } }, Collection: { acquireCount: { r: 2 } } } 106ms
2015-12-12T00:35:40.920+0000 I QUERY    [conn110692] query XXXX.endpoints query: { $query: { gw: { $gt: 0 }, $or: [ { status: "unmanaged" }, { status: "managed" } ] }, $readPreference: { mode: "secondaryPreferred" } } planSummary: IXSCAN { gw: -1.0, status: -1.0 } ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:0 keyUpdates:0 writeConflicts:0 numYields:3 nreturned:0 reslen:20 locks:{ Global: { acquireCount: { r: 8 } }, Database: { acquireCount: { r: 4 }, acquireWaitCount: { r: 4 }, timeAcquiringMicros: { r: 3690908386 } }, Collection: { acquireCount: { r: 4 } } } 182338ms
2015-12-12T00:35:40.927+0000 I COMMAND  [conn92491] command admin.$cmd command: listDatabases { listDatabases: 1 } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:290 locks:{ Global: { acquireCount: { r: 6 } }, Database: { acquireCount: { r: 3 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 14825915727 } } } 179101ms
2015-12-12T00:35:40.978+0000 E NETWORK  [conn92487] Uncaught std::exception: std::exception, terminating
2015-12-12T00:35:40.978+0000 I CONTROL  [conn92487] dbexit:  rc: 100

Comment by Githook User [ 12/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16429 Get rid of assertWriteLocked which was doing incomplete check
Branch: master
https://github.com/mongodb/mongo/commit/ab7a8649faf5c21901c719bbc88840ba7382a5d0

Comment by Githook User [ 12/Dec/14 ]

Author:

{u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}

Message: SERVER-16429: assert if locking not correct during oplog application
Branch: master
https://github.com/mongodb/mongo/commit/1b88ce0b469d85cff0575a6b0a98daa31a3f44e0

Comment by Githook User [ 10/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: Revert "SERVER-16429 Get rid of assertWriteLocked which was doing incomplete check"

This reverts commit 77f3100bc8753055224abb0c6439fc506eeb02de.
Branch: master
https://github.com/mongodb/mongo/commit/8e3fe7f38c3d1ae8e072839bd4e2aaf78e19783c

Comment by Githook User [ 10/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16429 Get rid of assertWriteLocked which was doing incomplete check
Branch: master
https://github.com/mongodb/mongo/commit/77f3100bc8753055224abb0c6439fc506eeb02de

Comment by Githook User [ 10/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16429 Add ScopedTransactions to places which were missing it
Branch: master
https://github.com/mongodb/mongo/commit/314f6a94d1c4993da47b00427ad2e9694653dc18

Comment by Githook User [ 10/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16429 Add ScopedTransaction to AutoGetCollectionForRead and Map-reduce
Branch: master
https://github.com/mongodb/mongo/commit/326d27c5fb249f9d372cb46fd8b97d725b31553a

Comment by Mathias Stearn [ 08/Dec/14 ]

This will be fixed by Kal's ScopedTransaction work.

Generated at Thu Feb 08 03:41:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.