[SERVER-21975] test_command.js failed in replset Created: 22/Oct/15  Updated: 18/Nov/16  Resolved: 04/Jan/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 3.2.5, 3.3.0

Type: Bug Priority: Major - P3
Reporter: Scott Hernandez (Inactive) Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: test-only
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-20979 add replication test commands to supp... Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:
Linked BF Score: 0

 Description   

Seems like an election is closing the connection causing it to fail:

failures

Example:

[js_test:test_command] 2015-10-22T16:00:19.896+0000 d20010| 2015-10-22T16:00:19.896+0000 I ASIO     [NetworkInterfaceASIO] Successfully connected to ip-10-65-78-2:20011
2015-10-22 16:00:21 +0000	
[js_test:test_command] 2015-10-22T16:00:21.894+0000 d20010| 2015-10-22T16:00:21.894+0000 I REPL     [ReplicationExecutor] Canceling priority takeover callback
[js_test:test_command] 2015-10-22T16:00:21.895+0000 d20010| 2015-10-22T16:00:21.894+0000 I REPL     [ReplicationExecutor] conducting a dry run election to see if we could be elected
[js_test:test_command] 2015-10-22T16:00:21.895+0000 d20010| 2015-10-22T16:00:21.895+0000 I REPL     [ReplicationExecutor] dry election run succeeded, running for election
[js_test:test_command] 2015-10-22T16:00:21.895+0000 d20011| 2015-10-22T16:00:21.895+0000 I REPL     [ReplicationExecutor] stepping down from primary, because a new term has begun: 2
[js_test:test_command] 2015-10-22T16:00:21.896+0000 d20011| 2015-10-22T16:00:21.895+0000 I REPL     [replExecDBWorker-1] transition to SECONDARY
[js_test:test_command] 2015-10-22T16:00:21.896+0000 d20011| 2015-10-22T16:00:21.895+0000 I NETWORK  [conn5] end connection 10.65.78.2:44160 (4 connections now open)
[js_test:test_command] 2015-10-22T16:00:21.896+0000 d20011| 2015-10-22T16:00:21.895+0000 I NETWORK  [conn1] end connection 127.0.0.1:59822 (4 connections now open)
[js_test:test_command] 2015-10-22T16:00:21.896+0000 d20010| 2015-10-22T16:00:21.896+0000 I REPL     [ReplicationExecutor] could not find member to sync from
[js_test:test_command] 2015-10-22T16:00:21.897+0000 d20011| 2015-10-22T16:00:21.896+0000 I NETWORK  [conn7] end connection 10.65.78.2:44172 (2 connections now open)
[js_test:test_command] 2015-10-22T16:00:21.897+0000 d20010| 2015-10-22T16:00:21.896+0000 I REPL     [ReplicationExecutor] election succeeded, assuming primary role in term 2
[js_test:test_command] 2015-10-22T16:00:21.897+0000 d20010| 2015-10-22T16:00:21.896+0000 I REPL     [ReplicationExecutor] transition to PRIMARY
[js_test:test_command] 2015-10-22T16:00:21.898+0000 d20010| 2015-10-22T16:00:21.896+0000 I COMMAND  [conn1] command admin.$cmd command: replSetTest { replSetTest: 1.0, waitForMemberState: 1.0, timeoutMillis: 60000.0 } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:22 locks:{} protocol:op_command 4974ms
[js_test:test_command] 2015-10-22T16:00:21.898+0000 d20011| 2015-10-22T16:00:21.896+0000 I NETWORK  [initandlisten] connection accepted from 10.65.78.2:44177 #9 (3 connections now open)
[js_test:test_command] 2015-10-22T16:00:21.898+0000 d20011| 2015-10-22T16:00:21.896+0000 I NETWORK  [conn3] end connection 10.65.78.2:44154 (2 connections now open)
[js_test:test_command] 2015-10-22T16:00:21.898+0000 2015-10-22T16:00:21.897+0000 E QUERY    [thread1] Error: error doing query: failed :
[js_test:test_command] 2015-10-22T16:00:21.899+0000 DB.prototype.runCommand@src/mongo/shell/db.js:132:1
[js_test:test_command] 2015-10-22T16:00:21.899+0000 Mongo.prototype.adminCommand@src/mongo/shell/mongo.js:58:12
[js_test:test_command] 2015-10-22T16:00:21.899+0000 @jstests/replsets/test_command.js:89:1
[js_test:test_command] 2015-10-22T16:00:21.899+0000 @jstests/replsets/test_command.js:5:2
[js_test:test_command] 2015-10-22T16:00:21.899+0000 
[js_test:test_command] 2015-10-22T16:00:21.899+0000 d20012| 2015-10-22T16:00:21.897+0000 I NETWORK  [initandlisten] connection accepted from 10.65.78.2:44263 #6 (4 connections now open)
[js_test:test_command] 2015-10-22T16:00:21.899+0000 failed to load: jstests/replsets/test_command.js



 Comments   
Comment by Githook User [ 25/Mar/16 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-21975 wait for secondary to become secondary before starting tests in replsets/test_command.js

change waitForMemberState to handle a potential network hangup in test_commands.js

Format JS code with approved style.

(cherry picked from commit adbe3cd2afe3aeda46c4576027eaa3482216f851)
(cherry picked from commit b8b3c3f5962aea2931df2b8807de35d2a9ff87aa)
Branch: v3.2
https://github.com/mongodb/mongo/commit/01c81a8a1e1635d4161b62b8ee445e16a24c5b09

Comment by Githook User [ 24/Mar/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: Revert "SERVER-21975 wait for secondary to become secondary before starting tests in replsets/test_command.js"

This reverts commit bec0b594ddde4d727b7598d46aebcefc4133bac5.
Branch: v3.2
https://github.com/mongodb/mongo/commit/452e759c5e4869998378a0a22871cee7af750042

Comment by Githook User [ 24/Mar/16 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-21975 wait for secondary to become secondary before starting tests in replsets/test_command.js

change waitForMemberState to handle a potential network hangup in test_commands.js

Format JS code with approved style.

(cherry picked from commit adbe3cd2afe3aeda46c4576027eaa3482216f851)
(cherry picked from commit b8b3c3f5962aea2931df2b8807de35d2a9ff87aa)
Branch: v3.2
https://github.com/mongodb/mongo/commit/bec0b594ddde4d727b7598d46aebcefc4133bac5

Comment by Andy Schwerin [ 24/Mar/16 ]

Approved for 3.2.5.

Comment by Benety Goh [ 24/Mar/16 ]

schwerin, can you approve for backport? This will fix BF-2045.

Comment by Githook User [ 04/Jan/16 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-21975 change waitForMemberState to handle a potential network hangup in test_commands.js
Branch: master
https://github.com/mongodb/mongo/commit/b8b3c3f5962aea2931df2b8807de35d2a9ff87aa

Comment by Matt Dannenberg [ 04/Jan/16 ]

Changing the assert.commandWorked() to an assert.soon() could fix the problem. CR imminent.

Comment by Githook User [ 21/Dec/15 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-21975 wait for secondary to become secondary before starting tests in replsets/test_command.js
Branch: master
https://github.com/mongodb/mongo/commit/adbe3cd2afe3aeda46c4576027eaa3482216f851

Comment by Adam Midvidy [ 21/Dec/15 ]

seems to have reoccurred:
https://evergreen.mongodb.com/task/mongodb_mongo_master_linux_64_debug_replicasets_auth_a3d8fbcadfeae8418ba17e5e90e51b929ff3ff93_15_12_18_22_40_14

Comment by Benety Goh [ 23/Oct/15 ]

Fixed in:

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-20979 make test_command.js more stable by waiting for node 0 to become primary before running tests
Branch: master
https://github.com/mongodb/mongo/commit/95344b7d99be4f7b30151d578b8b9b530c5be667

Generated at Thu Feb 08 03:59:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.