[SERVER-47109] Race in replset1.js from two-phase index build Created: 25/Mar/20  Updated: 29/Oct/23  Resolved: 06/Apr/20

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.2.6, 4.4.0-rc0, 4.7.0

Type: Bug Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: A. Jesse Jiryu Davis
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
is caused by SERVER-37270 Remove foreground index build functio... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4, v4.2
Participants:
Linked BF Score: 17

 Description   

In replsettest1.js we do:

    t.ensureIndex({a: 1});
 
    var result = db.runCommand({getLastError: 1, w: 3, wtimeout: 30000});
    printjson(result);
    var lastOp = result.lastOp;
    const oplogEntries = replTest.dumpOplog(master);
    const lastOplogOp = oplogEntries[0];
    assert.eq(lastOplogOp['ts'], lastOp['ts']);
    assert.eq(lastOplogOp['t'], lastOp['t']);

Between the getLastError and the dumpOplog, one more oplog entry was written. The last two oplog entries are (newest first):

{  "op" : "u",  "ns" : "config.system.indexBuilds",  "ui" : UUID("1d972c54-1ce0-4789-ab21-bcfdd02897ce"),  "o" : {  "$v" : 1,  "$set" : {  "commitReadyMembers" : [ "ip-10-122-74-53:21272", "ip-10-122-74-53:21270", "ip-10-122-74-53:21271" ] } },  "o2" : {  "_id" : UUID("0695d117-05b1-4883-a949-3950b0da1770") },  "ts" : Timestamp(1584664055, 6),  "t" : NumberLong(2),  "wall" : ISODate("2020-03-20T00:27:35.668Z"),  "v" : NumberLong(2) }
{  "op" : "c",  "ns" : "foo.$cmd",  "ui" : UUID("943c7b6a-a1dd-48e0-aeb5-3c9a6142ddf6"),  "o" : {  "commitIndexBuild" : "foo",  "indexBuildUUID" : UUID("0695d117-05b1-4883-a949-3950b0da1770"),  "indexes" : [ { "v" : 2, "key" : { "a" : 1 }, "name" : "a_1" } ] },  "ts" : Timestamp(1584664055, 5),  "t" : NumberLong(2),  "wall" : ISODate("2020-03-20T00:27:35.660Z"),  "v" : NumberLong(2) }

The test expects no new oplog entries between getLastError and dumpOplog, so it fails:

assert: [Timestamp(1584664055, 6)] != [Timestamp(1584664055, 5)] are not equal
doassert@src/mongo/shell/assert.js:20:14
assert.eq@src/mongo/shell/assert.js:179:9
doTest@jstests/replsets/replset1.js:133:5

We probably need to wait for the last oplog entry resulting from the ensureIndex call before calling getLastError. Handing off to Execution since this appears to be an intermittent failure from "Simultaneous indexing on all nodes (two phase commit)".



 Comments   
Comment by Githook User [ 06/Apr/20 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: SERVER-47109 Fix index-build race in replset1.js, try 2

(cherry picked from commit 188862afbe0c75b09e6dddd56e7f35708d5c4df9)
Branch: v4.4
https://github.com/mongodb/mongo/commit/b6cac93c3a10a85039f63b16f981020d555f20be

Comment by Githook User [ 06/Apr/20 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: SERVER-47109 Fix index-build race in replset1.js, try 2
Branch: master
https://github.com/mongodb/mongo/commit/188862afbe0c75b09e6dddd56e7f35708d5c4df9

Comment by Githook User [ 30/Mar/20 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: SERVER-47109 Fix index-build race in replset1.js

(cherry picked from commit c0c26887102d86cc887cbac642b441af896ba3cb)
Branch: v4.4
https://github.com/mongodb/mongo/commit/1203f5e760a1f9d62c0742e43a91d0ea1789e778

Comment by Githook User [ 25/Mar/20 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: SERVER-47109 Fix index-build race in replset1.js

(cherry picked from commit c0c26887102d86cc887cbac642b441af896ba3cb)
Branch: v4.2
https://github.com/mongodb/mongo/commit/771ccef97fe72078db59323397530572f33613e2

Comment by Githook User [ 25/Mar/20 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: SERVER-47109 Fix index-build race in replset1.js
Branch: master
https://github.com/mongodb/mongo/commit/c0c26887102d86cc887cbac642b441af896ba3cb

Generated at Thu Feb 08 05:13:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.