-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.0.8
-
Component/s: Sharding, Stability, WiredTiger
-
Labels:None
-
ALL
-
Mongod crashed on 1st config server after removal of a shard from a cluster.
MongoDB version: 3.0.8.
Log file on config server shows:
2016-10-13T22:13:38.506+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2016-10-13T22:13:38.506+0000 I CONTROL [signalProcessingThread] now exiting
2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets...
2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] closing listening socket: 6
2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] closing listening socket: 7
2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog...
2016-10-13T22:13:38.506+0000 I NETWORK [signalProcessingThread] shutdown: going to close sockets...
2016-10-13T22:13:38.514+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2016-10-13T22:13:38.532+0000 I STORAGE [conn12551] got request after shutdown()
2016-10-13T22:13:38.533+0000 I STORAGE [conn12550] got request after shutdown()
2016-10-13T22:13:38.617+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock...
2016-10-13T22:13:38.617+0000 I CONTROL [signalProcessingThread] dbexit: rc: 0
Restarting the service failed with permissions errors. It seems like the ownership of few files was changed from mongod:mongod to root:root during or before the crash. Some kind of bug in mongod? I don't think anyone changed the ownership.
These are the files with wrong ownership:
rw-rr- 1 root root 5701900 Oct 13 22:13 mongod.log
rw-rr- 1 root root 913 Oct 13 22:13 WiredTiger.turtle
rw-rr- 1 root root 64598272 Oct 13 22:13 WiredTigerLog.0000000452
This type of crash happened before, where 1st config server crashed after shard removal, but I do not recall if ownership was also broken.