Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32398

Primary freezes during background index build

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Duplicate
    • 3.4.10
    • None
    • Ubuntu 16.04 Server
    • ALL

    Description

      mongod 3.4.10 on Ubuntu 16.04 in a replica set with 3 nodes. The primary was building an index and the index build stopped:

      Dec 18 22:05:57 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4027500/10026608 40%
      Dec 18 22:06:00 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4062300/10026608 40%
      Dec 18 22:06:03 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4096100/10026608 40%
      Dec 18 22:06:06 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4127300/10026608 41%
      Dec 18 22:06:09 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4152100/10026608 41%
      Dec 18 22:06:12 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4189100/10026608 41%
      Dec 18 22:06:15 hostname mongod.27017[1250]: [conn5562]   Index Build (background): 4241600/10026608 42%
      

      At this point it stalled, and no more "Index Build" lines were printed until I restarted mongod several hours later. From here on I could connect to the admin database as the root database user, but I couldn't connect as any other user - the connection would just hang. Logs below in case they're relevant (I suspect they're not).

      Dec 18 22:06:38 hostname monit[1400]: 'mongo_index_size' '/usr/local/bin/mongo_index_size.sh' failed with exit status (1) -- Mongo indexes are consuming 14900 MB, higher than threshold of 12034 MB
      Dec 18 22:06:38 hostname mongod.27017[1250]: [thread1] connection accepted from 127.0.0.1:45244 #5605 (66 connections now open)
      Dec 18 22:06:38 hostname mongod.27017[1250]: [thread1] connection accepted from 127.0.0.1:45246 #5606 (67 connections now open)
      Dec 18 22:06:38 hostname mongod.27017[1250]: [conn5606] received client metadata from 127.0.0.1:45246 conn5606: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
      Dec 18 22:06:38 hostname mongod.27017[1250]: [conn5605] received client metadata from 127.0.0.1:45244 conn5605: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
      

      Backtrace attached.

      Attachments

        1. gdb-mongod-2.txt
          735 kB
        2. unique_threads.txt
          133 kB

        Issue Links

          Activity

            People

              kelsey.schubert@mongodb.com Kelsey Schubert
              mzs Michael Smith
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: