Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26627

Config server crash after shard removal

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.8
    • Component/s: Sharding, Stability, WiredTiger
    • Labels:
      None
    • ALL
    • Hide

      Not sure if it is easily reproducible:
      Remove shard from 2 shard cluster.

      Show
      Not sure if it is easily reproducible: Remove shard from 2 shard cluster.

      Mongod crashed on 1st config server after removal of a shard from a cluster.

      MongoDB version: 3.0.8.

      Log file on config server shows:
      2016-10-13T22:13:38.506+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
      2016-10-13T22:13:38.506+0000 I CONTROL [signalProcessingThread] now exiting
      2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets...
      2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] closing listening socket: 6
      2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] closing listening socket: 7
      2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
      2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog...
      2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] shutdown: going to close sockets...
      2016-10-13T22:13:38.514+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
      2016-10-13T22:13:38.532+0000 I STORAGE [conn12551] got request after shutdown()
      2016-10-13T22:13:38.533+0000 I STORAGE [conn12550] got request after shutdown()
      2016-10-13T22:13:38.617+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock...
      2016-10-13T22:13:38.617+0000 I CONTROL [signalProcessingThread] dbexit: rc: 0

      Restarting the service failed with permissions errors. It seems like the ownership of few files was changed from mongod:mongod to root:root during or before the crash. Some kind of bug in mongod? I don't think anyone changed the ownership.

      These are the files with wrong ownership:
      rw-rr- 1 root root 5701900 Oct 13 22:13 mongod.log
      rw-rr- 1 root root 913 Oct 13 22:13 WiredTiger.turtle
      rw-rr- 1 root root 64598272 Oct 13 22:13 WiredTigerLog.0000000452

      This type of crash happened before, where 1st config server crashed after shard removal, but I do not recall if ownership was also broken.

            Assignee:
            Unassigned Unassigned
            Reporter:
            dabest1 dabest1
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: