Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32398

Primary freezes during background index build

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.10
    • Component/s: Index Maintenance, WiredTiger
    • Environment:
      Ubuntu 16.04 Server
    • ALL

      mongod 3.4.10 on Ubuntu 16.04 in a replica set with 3 nodes. The primary was building an index and the index build stopped:

      Dec 18 22:05:57 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4027500/10026608 40%
Dec 18 22:06:00 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4062300/10026608 40%
Dec 18 22:06:03 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4096100/10026608 40%
Dec 18 22:06:06 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4127300/10026608 41%
Dec 18 22:06:09 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4152100/10026608 41%
Dec 18 22:06:12 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4189100/10026608 41%
Dec 18 22:06:15 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4241600/10026608 42%

      At this point it stalled, and no more "Index Build" lines were printed until I restarted mongod several hours later. From here on I could connect to the admin database as the root database user, but I couldn't connect as any other user - the connection would just hang. Logs below in case they're relevant (I suspect they're not).

      Dec 18 22:06:38 hostname monit[1400]: 'mongo_index_size' '/usr/local/bin/mongo_index_size.sh' failed with exit status (1) -- Mongo indexes are consuming 14900 MB, higher than threshold of 12034 MB
Dec 18 22:06:38 hostname mongod.27017[1250]: [thread1] connection accepted from 127.0.0.1:45244 #5605 (66 connections now open)
Dec 18 22:06:38 hostname mongod.27017[1250]: [thread1] connection accepted from 127.0.0.1:45246 #5606 (67 connections now open)
Dec 18 22:06:38 hostname mongod.27017[1250]: [conn5606] received client metadata from 127.0.0.1:45246 conn5606: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
Dec 18 22:06:38 hostname mongod.27017[1250]: [conn5605] received client metadata from 127.0.0.1:45244 conn5605: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }

      Backtrace attached.

        1. gdb-mongod-2.txt
          735 kB
          Michael Smith
        2. unique_threads.txt
          133 kB
          Kelsey Schubert

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            mzs Michael Smith
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: