Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38084

MongoDB hangs when a part of a replica set

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 3.6.2
    • Replication
    • None
    • ALL

    Description

      We have been running a single instance of Mongo for a number of years now with our application without any issues at all.

      We have now set up a replica set with 3 members: 1 primary, 1 secondary and an arbiter.

      After a couple of hours our application simply stops working with the database i.e. it hangs. Below is an extract from the database log file from the primary server. In this log the activity seems to stop at 14:09:15 with our application (running on 192.168.45.105). Address 192.168.45.106 is the secondary server. All we get after that is successful authentication and nothing else. If we restart our application it stops on the first database call. If Mongo is restarted on the primary and secondary servers and then our application restarted it works again for another couple of hours and the cycle repeats.

      2018-11-09T14:09:14.973+0000 I COMMAND  [conn254] command map.$cmd command: update \{ update: "items", writeConcern: { w: 1 }, ordered: true, $db: "map" } numYields:0 reslen:119 locks:\{ Global: { acquireCount: { r: 3, w: 3 } }, Database: \{ acquireCount: { w: 3 } }, Collection: \{ acquireCount: { w: 2 } }, oplog: \{ acquireCount: { w: 1 } } } protocol:op_query 199ms
       
      2018-11-09T14:09:14.991+0000 I COMMAND  [conn260] command map.data command: find \{ find: "data", filter: { _id: ObjectId('5afdb5c1238164595c201b71') }, limit: 1, singleBatch: true, $db: "map" } planSummary: IDHACK keysExamined:1 docsExamined:1 cursorExhausted:1 numYields:1 nreturned:1 reslen:7697 locks:\{ Global: { acquireCount: { r: 4 } }, Database: \{ acquireCount: { r: 2 } }, Collection: \{ acquireCount: { r: 2 } } } protocol:op_query 116ms
       
      2018-11-09T14:09:15.002+0000 I COMMAND  [conn269] command map.data command: find \{ find: "data", filter: { FName: "aw pesticide\rainfall\geometry types\subcatchment\cat ditch\projected grid" }, sort: \{ _id: -1 }, skip: 0, limit: 1, noCursorTimeout: false, $db: "map" } planSummary: IXSCAN \{ FName: 1 } keysExamined:1 docsExamined:1 hasSortStage:1 cursorExhausted:1 numYields:1 nreturned:1 reslen:2688 locks:\{ Global: { acquireCount: { r: 4 } }, Database: \{ acquireCount: { r: 2 } }, Collection: \{ acquireCount: { r: 2 } } } protocol:op_query 126ms
       
      2018-11-09T14:09:24.936+0000 I NETWORK  [listener] connection accepted from [192.168.45.106:49579|http://192.168.45.106:49579/] #328 (70 connections now open)
       
      2018-11-09T14:09:24.937+0000 I NETWORK  [conn328] received client metadata from [192.168.45.106:49579|http://192.168.45.106:49579/] conn: \{ driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.2" }, os: \{ type: "Windows", name: "Microsoft Windows Server 2012 R2", architecture: "x86_64", version: "6.3 (build 9600)" } }
       
      2018-11-09T14:09:24.943+0000 I ACCESS   [conn328] Successfully authenticated as principal __system on local
       
      2018-11-09T14:09:33.290+0000 I NETWORK  [listener] connection accepted from [192.168.45.105:60568|http://192.168.45.105:60568/] #329 (71 connections now open)
       
      2018-11-09T14:09:33.292+0000 I NETWORK  [conn329] received client metadata from [192.168.45.105:60568|http://192.168.45.105:60568/] conn: \{ driver: { name: "mongo-csharp-driver", version: "2.4.4.0" }, os: \{ type: "Windows", name: "Microsoft Windows", architecture: "x86_64" }, platform: ".NET Framework 4.7.3163.0" }
       
      2018-11-09T14:09:33.428+0000 I ACCESS   [conn329] Successfully authenticated as principal menadmin on admin
       
      2018-11-09T14:10:01.028+0000 I NETWORK  [listener] connection accepted from [192.168.45.105:60570|http://192.168.45.105:60570/] #330 (72 connections now open)
       
      2018-11-09T14:10:01.029+0000 I NETWORK  [conn330] received client metadata from [192.168.45.105:60570|http://192.168.45.105:60570/] conn: \{ driver: { name: "mongo-csharp-driver", version: "2.4.4.0" }, os: \{ type: "Windows", name: "Microsoft Windows", architecture: "x86_64" }, platform: ".NET Framework 4.7.3163.0" }
       
      2018-11-09T14:10:01.201+0000 I ACCESS   [conn330] Successfully authenticated as principal menadmin on admin
       
      2018-11-09T14:10:09.791+0000 I NETWORK  [listener] connection accepted from [192.168.45.105:60571|http://192.168.45.105:60571/] #331 (73 connections now open)
       
      2018-11-09T14:10:09.793+0000 I NETWORK  [conn331] received client metadata from [192.168.45.105:60571|http://192.168.45.105:60571/] conn: \{ driver: { name: "mongo-csharp-driver", version: "2.4.4.0" }, os: \{ type: "Windows", name: "Microsoft Windows", architecture: "x86_64" }, platform: ".NET Framework 4.7.3163.0" }
       
      2018-11-09T14:10:09.922+0000 I ACCESS   [conn331] Successfully authenticated as principal menadmin on admin
       
      2018-11-09T14:10:29.935+0000 I NETWORK  [listener] connection accepted from [192.168.45.106:49580|http://192.168.45.106:49580/] #332 (74 connections now open)
       
      2018-11-09T14:10:29.936+0000 I NETWORK  [conn332] received client metadata from [192.168.45.106:49580|http://192.168.45.106:49580/] conn: \{ driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.2" }, os: \{ type: "Windows", name: "Microsoft Windows Server 2012 R2", architecture: "x86_64", version: "6.3 (build 9600)" } }
       
      2018-11-09T14:10:29.941+0000 I ACCESS   [conn332] Successfully authenticated as principal __system on local
       
      2018-11-09T14:11:34.935+0000 I NETWORK  [listener] connection accepted from [192.168.45.106:49581|http://192.168.45.106:49581/] #333 (75 connections now open)
       
      2018-11-09T14:11:34.936+0000 I NETWORK  [conn333] received client metadata from [192.168.45.106:49581|http://192.168.45.106:49581/] conn: \{ driver: { name: "NetworkInterfaceASIO-RS", version: "3.6.2" }, os: \{ type: "Windows", name: "Microsoft Windows Server 2012 R2", architecture: "x86_64", version: "6.3 (build 9600)" } }
       
      2018-11-09T14:11:34.942+0000 I ACCESS   [conn333] Successfully authenticated as principal __system on local
       
      2018-11-09T14:11:51.225+0000 I NETWORK  [listener] connection accepted from [192.168.45.105:60577|http://192.168.45.105:60577/] #334 (76 connections now open)
       
      2018-11-09T14:11:51.226+0000 I NETWORK  [conn334] received client metadata from [192.168.45.105:60577|http://192.168.45.105:60577/] conn: \{ driver: { name: "mongo-csharp-driver", version: "2.4.4.0" }, os: \{ type: "Windows", name: "Microsoft Windows", architecture: "x86_64" }, platform: ".NET Framework 4.7.3163.0" }
       
      2018-11-09T14:11:51.368+0000 I ACCESS   [conn334] Successfully authenticated as principal menadmin on admin
       
      2018-11-09T14:39:30.813+0000 I NETWORK  [listener] connection accepted from [192.168.45.105:60671|http://192.168.45.105:60671/] #335 (77 connections now open)
       
      2018-11-09T14:39:30.814+0000 I NETWORK  [conn335] received client metadata from [192.168.45.105:60671|http://192.168.45.105:60671/] conn: \{ driver: { name: "mongo-csharp-driver", version: "2.4.4.0" }, os: \{ type: "Windows", name: "Microsoft Windows", architecture: "x86_64" }, platform: ".NET Framework 4.7.3163.0" }
       
      2018-11-09T14:39:30.951+0000 I ACCESS   [conn335] Successfully authenticated as principal menadmin on admin
       
      2018-11-09T16:13:03.502+0000 I CONTROL  [serviceShutdown] got SERVICE_CONTROL_STOP request from Windows Service Control Manager, will terminate after current cmd ends
      

       

      Attachments

        1. error - mongo.png
          error - mongo.png
          183 kB
        2. metrics.zip
          40.66 MB
        3. metrics.zip
          20.68 MB
        4. mongo.log
          47 kB
        5. mongo 2.txt
          181 kB

        Activity

          People

            daniel.hatcher@mongodb.com Danny Hatcher (Inactive)
            Sputink1969 Ian Hannah [X]
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: