[SERVER-16027] debug intermittent failure of read_pref_multi_mongos_stale_config.js Created: 07/Nov/14  Updated: 11/Jul/16  Resolved: 11/Nov/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 2.8.0-rc0

Type: Task Priority: Major - P3
Reporter: Greg Studer Assignee: Greg Studer
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Participants:
Linked BF Score: 0

 Description   

Intermittently the read_pref_multi_mongos_stale_config.js test fails across different 2.7 builds.

https://mci.10gen.com/task/mongodb_mongo_master_linux_64_debug_716cfde34adf0e1c4a92c4e8fc1960a50d3cbc50_14_11_07_15_40_07_sharding_linux_64_debug



 Comments   
Comment by Greg Studer [ 11/Nov/14 ]

Issue was the replica set monitor wasn't getting correctly initialized before the writes occurred and the test did not verify that the writes actually succeeded. Autosplit generally initialized the replica set monitor at some point, but when autosplit wasn't triggered (because of random initialization of the thresholds) certain documents would then not exist and the test would later fail.

Comment by Githook User [ 11/Nov/14 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-16027 create replica set monitor when req'd when resolving shards for write
Branch: master
https://github.com/mongodb/mongo/commit/6b73c5c63de972ebd39ab85c93f81e7040fe6950

Comment by Greg Studer [ 11/Nov/14 ]

Repro'd:
https://mci.10gen.com/task/mongodb_mongo_master_linux_64_debug_716cfde34adf0e1c4a92c4e8fc1960a50d3cbc50_14_11_07_15_40_07_sharding_linux_64_debug

Seems like the issue is mongod not returning any data to mongos, not clear if the data is getting filtered by bad metadata or something more sinister is going on:

 m30999| 2014-11-10T22:05:50.576+0000 D NETWORK  [conn1] finishing on shard test-rs1:test-rs1/EC2AMAZ-ZVRDQB5:31200,EC2AMAZ-ZVRDQB5:31201,EC2AMAZ-ZVRDQB5:31202, current connection state is { state: { conn: "test-rs1/EC2AMAZ-ZVRDQB5:31200,EC2AMAZ-ZVRDQB5:31201,EC2AMAZ-ZVRDQB5:31202", vinfo: "test.user @ 2|1||546136bb628dd10c6d520fc5", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: false, errored: false }
 m30999| 2014-11-10T22:05:50.577+0000 D NETWORK  [conn1] finished on shard test-rs1:test-rs1/EC2AMAZ-ZVRDQB5:31200,EC2AMAZ-ZVRDQB5:31201,EC2AMAZ-ZVRDQB5:31202, current connection state is { state: { conn: "(done)", vinfo: "test.user @ 2|1||546136bb628dd10c6d520fc5", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false }
^^^^Note the empty cursor
 m30999| 2014-11-10T22:05:50.577+0000 I NETWORK  [conn1] PCursor erasing empty state { state: {}, retryNext: false, init: false, finish: false, errored: false }
assert failed
Error: assert failed
    at Error (<anonymous>)
    at doassert (src/mongo/shell/assert.js:11:14)
    at assert (src/mongo/shell/assert.js:20:5)
    at D:\data\mci\shell\src\jstests\sharding\read_pref_multi_mongos_stale_config.js:31:1
2014-11-10T22:05:50.579+0000 I QUERY    Error: assert failed
    at Error (<anonymous>)
    at doassert (src/mongo/shell/assert.js:11:14)
    at assert (src/mongo/shell/assert.js:20:5)
    at D:\data\mci\shell\src\jstests\sharding\read_pref_multi_mongos_stale_config.js:31:1 at src/mongo/shell/assert.js:13
failed to load: D:\data\mci\shell\src\jstests\sharding\read_pref_multi_mongos_stale_config.js

Comment by Greg Studer [ 10/Nov/14 ]

Has not reproduced since 11/7 - https://github.com/mongodb/mongo/commit/001d34a6a7bc13c3d6d159d7149417f5e6571359 - waiting for more information.

Comment by Githook User [ 07/Nov/14 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-16027 verbose mongos cursor messages for read_pref_multi_mongos_stale_config.js
Branch: master
https://github.com/mongodb/mongo/commit/001d34a6a7bc13c3d6d159d7149417f5e6571359

Generated at Thu Feb 08 03:39:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.