Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21293

Loading config data when CSRS member is blackholed gets mongos stuck

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.2.0-rc4
    • Affects Version/s: 3.2.0-rc2
    • Component/s: Sharding
    • None
    • Fully Compatible
    • ALL
    • Hide 
      python buildscripts/resmoke.py --executor sharding blackhole_first_config_server_from_mongos.js
      Show
      python buildscripts/resmoke.py --executor sharding blackhole_first_config_server_from_mongos.js
    • Sharding C (11/20/15)

      When the following sequence of events happen, mongos seems to retry reading the chunk data infinitely from the first config server:
      1. Initialize the sharded cluster
      2. Enable sharding on the "test" database
      3. Create a sharded collection called "server16691" with shard key {_id: 1}
      4. Configure the mongobridge corresponding to the first config server to discard messages from the mongos
      5. Configure the mongobridges corresponding to the other config servers to delay messages from the mongos
      6. Run the "flushRouterConfig" command to clear the routing table of the mongos
      7. Try to run db.server16691.find({}).itcount() to trigger read operations of the chunk data

      [js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:35:57.238-0500 b20014| 2015-11-04T14:35:57.229-0500 I -        [thread2] Received "discardMessagesFrom" command with arguments { discardMessagesFrom: 1.0, $forBridge: true, host: "hanamizu:20020", loss: 1.0 } from <unknown>
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:35:57.240-0500 b20014| 2015-11-04T14:35:57.231-0500 I -        [thread12] Received "find" command with arguments { find: "databases", filter: { _id: "test" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1446665757000|20, t: 1 } }, limit: 1, maxTimeMS: 30000 } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:35:57.240-0500 b20014| 2015-11-04T14:35:57.231-0500 I -        [thread12] Discarding "find" command with arguments { find: "databases", filter: { _id: "test" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1446665757000|20, t: 1 } }, limit: 1, maxTimeMS: 30000 } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:07.007-0500 b20014| 2015-11-04T14:36:07.007-0500 I -        [thread24] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:07.008-0500 b20014| 2015-11-04T14:36:07.007-0500 I -        [thread24] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:13.883-0500 b20014| 2015-11-04T14:36:13.883-0500 I -        [thread11] Received "ismaster" command with arguments { ismaster: 1 } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:13.883-0500 b20014| 2015-11-04T14:36:13.883-0500 I -        [thread11] Discarding "ismaster" command with arguments { ismaster: 1 } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:26.807-0500 b20014| 2015-11-04T14:36:26.807-0500 I -        [thread25] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:26.808-0500 b20014| 2015-11-04T14:36:26.807-0500 I -        [thread25] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:41.806-0500 b20014| 2015-11-04T14:36:41.806-0500 I -        [thread26] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:41.807-0500 b20014| 2015-11-04T14:36:41.806-0500 I -        [thread26] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:57.008-0500 b20014| 2015-11-04T14:36:57.007-0500 I -        [thread27] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:36:57.008-0500 b20014| 2015-11-04T14:36:57.007-0500 I -        [thread27] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
...
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:38:57.608-0500 b20014| 2015-11-04T14:38:57.607-0500 I -        [thread38] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:38:57.608-0500 b20014| 2015-11-04T14:38:57.607-0500 I -        [thread38] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:12.808-0500 b20014| 2015-11-04T14:39:12.808-0500 I -        [thread39] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:12.808-0500 b20014| 2015-11-04T14:39:12.808-0500 I -        [thread39] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:22.808-0500 b20014| 2015-11-04T14:39:22.807-0500 I -        [thread40] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:22.808-0500 b20014| 2015-11-04T14:39:22.807-0500 I -        [thread40] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:28.008-0500 b20014| 2015-11-04T14:39:28.007-0500 I -        [thread41] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:28.008-0500 b20014| 2015-11-04T14:39:28.007-0500 I -        [thread41] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:43.207-0500 b20014| 2015-11-04T14:39:43.207-0500 I -        [thread42] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:43.208-0500 b20014| 2015-11-04T14:39:43.207-0500 I -        [thread42] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:58.008-0500 b20014| 2015-11-04T14:39:58.007-0500 I -        [thread43] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:39:58.008-0500 b20014| 2015-11-04T14:39:58.007-0500 I -        [thread43] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:40:13.006-0500 b20014| 2015-11-04T14:40:13.006-0500 I -        [thread44] Received "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020
[js_test:blackhole_first_config_server_from_mongos] 2015-11-04T14:40:13.006-0500 b20014| 2015-11-04T14:40:13.006-0500 I -        [thread44] Discarding "isMaster" command with arguments { isMaster: 1, hostInfo: "hanamizu:20020" } from hanamizu:20020

        1. test_output.log
          1002 kB
        2. stacks.log
          24 kB
        3. blackhole_first_config_server_from_mongos.js
          2 kB

            Assignee:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: