Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20729

Database with WT fails to start after clean shutdown (missing WiredTiger.wt)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 3.0.2, 3.0.3, 3.0.4
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • Environment:
      Linux 3.13.0-65-generic #105-Ubuntu SMP Mon Sep 21 18:50:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      1. Start mongodb with WT (sudo service mongod start)
      2. Stop mongodb (sudo service mongod stop) - repeat 1,2 many times.
      3. Logs will indicate issue and mongodb will fail to start.

      Show
      1. Start mongodb with WT (sudo service mongod start) 2. Stop mongodb (sudo service mongod stop) - repeat 1,2 many times. 3. Logs will indicate issue and mongodb will fail to start.

      Description

      This is not a file permission issue. Somehow after a clean shutdown the WiredTiger.wt file is missing and this means you cannot start mongodb again, effectively losing the database.

      I have seen this a few times since using wired tiger, on 3.0.2 and currently on 3.0.4

      This is out /etc/fstab for /data partition:
      `/dev/xvdb /data xfs noatime 0 0`

      Here is the mongodb log:

      Oct  1 11:04:10 ip-172-31-0-19 mongod.27017[1127]: [conn20] build index done.  scanned 292 total records. 0 secs
      Oct  1 11:04:10 ip-172-31-0-19 mongod.27017[1127]: [conn19] end connection 127.0.0.1:45601 (5 connections now open)
      Oct  1 11:04:10 ip-172-31-0-19 mongod.27017[1127]: [conn21] end connection 127.0.0.1:45603 (5 connections now open)
      Oct  1 11:04:10 ip-172-31-0-19 mongod.27017[1127]: [conn22] end connection 127.0.0.1:45604 (5 connections now open)
      Oct  1 11:04:10 ip-172-31-0-19 mongod.27017[1127]: [conn20] end connection 127.0.0.1:45602 (2 connections now open)
      Oct  1 11:04:39 ip-172-31-0-19 mongod.27017[1127]: [initandlisten] connection accepted from 127.0.0.1:45606 #23 (3 connections now open)
      Oct  1 11:04:39 ip-172-31-0-19 mongod.27017[1127]: [conn23] end connection 127.0.0.1:45606 (2 connections now open)
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] now exiting
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] shutdown: going to close listening sockets...
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] closing listening socket: 5
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] closing listening socket: 6
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] shutdown: going to flush diaglog...
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] shutdown: going to close sockets...
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] WiredTigerKVEngine shutting down
      Oct  1 16:00:04 alpha mongod.27017[1127]: [conn18] end connection 127.0.0.1:45239 (1 connection now open)
      Oct  1 16:00:04 alpha mongod.27017[1127]: [conn17] end connection 127.0.0.1:45229 (1 connection now open)
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] shutdown: removing fs lock...
      Oct  1 16:00:04 alpha mongod.27017[1127]: [signalProcessingThread] dbexit:  rc: 0
      Oct  1 20:00:44 alpha mongod.27017[1074]: [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      Oct  1 20:00:44 alpha mongod.27017[1074]: [initandlisten] WiredTiger (2) [1443729644:306783][1074:0x7fdba8fd4bc0], file:WiredTiger.wt, connection: /data/db/WiredTiger.wt: No such file or directory
      Oct  1 20:00:44 alpha mongod.27017[1074]: [initandlisten] Assertion: 28595:2: No such file or directory
      Oct  1 20:00:44 alpha mongod.27017[1074]: [initandlisten] exception in initAndListen: 28595 2: No such file or directory, terminating
      Oct  1 20:00:44 alpha mongod.27017[1074]: [initandlisten] dbexit:  rc: 100
      

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: