db on mounted GlusterFS remote filesystem causes kernel panic (task stuck for 120 seconds) during mongod shutdown

XMLWordPrintableJSON

    • Type: Question
    • Resolution: Incomplete
    • Priority: Major - P3
    • None
    • Affects Version/s: 3.6.5
    • Component/s: None
    • None
    • Environment:
      Kubernetes 1.9.5
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      Running a mongo pod on Kubernetes (6 CentOS 7.4 nodes) with preStop command ["mongod", "–shutdown"] and the db volume is mounted on a remote GlusterFS volume (with a subdir-mount option) using a PersistentVolume and a PV-Claim for the mongo deployment. The Gluster volume is striped onto 3 hosts without replicas and is mounted using the user-space (FUSE) client. All the nodes and the gluster hosts are setup to NTP their time from the same time servers.

      When the remote mount is empty, mongo starts, populates the directory, operates without errors and shuts down without errors.

      On second start, mongo will start without errors and operates without errors. On shutdown, mongo will log the event that WiredTiger is shutting down and after two minutes the host's kernel will panic with task mongod stuck for more than 120 seconds.

      3.6.0 using MMAPv1 works as intended. I previously had problems using GlusterFS volumes with replication for Elasticsearch storage (regarding the creation and access times on lock files), but none with striped volumes.

            Assignee:
            Ramon Fernandez Marina
            Reporter:
            Ferhat Savc?
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: