Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-6937

WT_PANIC: WiredTiger library panic on mongod --repair

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.24
    • Component/s: None
    • None
    • Environment:
      NetBSD 9.1_STABLE amd64

      Unifi mongodb database failed to startup with error.

       

      2020-11-22T16:04:09.349+0000 I CONTROL [initandlisten] MongoDB starting : pid=17483 port=27117 dbpath=/usr/pkg/unifi/data/db 64-bit host=iris.absd.org
      2020-11-22T16:04:09.350+0000 I CONTROL [initandlisten] db version v3.4.24
      2020-11-22T16:04:09.350+0000 I CONTROL [initandlisten] git version: 865b4f6a96d0f5425e39a18337105f33e8db504d
      2020-11-22T16:04:09.350+0000 I CONTROL [initandlisten] allocator: system
      2020-11-22T16:04:09.351+0000 I CONTROL [initandlisten] modules: none
      2020-11-22T16:04:09.351+0000 I CONTROL [initandlisten] build environment:
      2020-11-22T16:04:09.351+0000 I CONTROL [initandlisten] distarch: x86_64
      2020-11-22T16:04:09.352+0000 I CONTROL [initandlisten] target_arch: x86_64
      2020-11-22T16:04:09.352+0000 I CONTROL [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27117, unixDomainSocket: { pathPrefix: "/usr/pkg/unifi/run" } }, processManagement: { pidFilePath: "/usr/pkg/unifi/run/mongod.pid" }, storage: { dbPath: "/usr/pkg/unifi/data/db" }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/usr/pkg/unifi/logs/mongod.log" } }
      2020-11-22T16:04:09.353+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1535M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
      2020-11-22T16:11:03.227+0000 E STORAGE [initandlisten] WiredTiger error (0) [1606061463:227580][17483:0x71c2ee70d800], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 20480B block at offset 212992: block header checksum of 4113582198 doesn't match expected checksum of 104923011
      2020-11-22T16:11:03.228+0000 E STORAGE [initandlisten] WiredTiger error (0) [1606061463:228110][17483:0x71c2ee70d800], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value
      2020-11-22T16:11:03.228+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1606061463:228612][17483:0x71c2ee70d800], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2020-11-22T16:11:03.229+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
      2020-11-22T16:11:03.229+0000 I - [initandlisten]
      ***aborting after fassert() failure
      
      2020-11-22T16:11:03.464+0000 F - [initandlisten] Got signal: 6 (Abort trap).
      

       

      Attempting to run repair:

      bin/mongod --dbpath /usr/pkg/unifi/data/db --repair

      runs for some time then exist with "WiredTiger.wt: encountered an illegal file format or internal value" then "the process must exit and restart: WT_PANIC: WiredTiger library panic"

       

      2020-11-22T17:36:52.669+0000 I CONTROL [initandlisten] MongoDB starting : pid=922 port=27017 dbpath=/usr/pkg/unifi/data/db 64-bit host=iris.absd.org
      2020-11-22T17:36:52.673+0000 I CONTROL [initandlisten] db version v3.4.24
      2020-11-22T17:36:52.673+0000 I CONTROL [initandlisten] git version: 865b4f6a96d0f5425e39a18337105f33e8db504d
      2020-11-22T17:36:52.673+0000 I CONTROL [initandlisten] allocator: system
      2020-11-22T17:36:52.674+0000 I CONTROL [initandlisten] modules: none
      2020-11-22T17:36:52.674+0000 I CONTROL [initandlisten] build environment:
      2020-11-22T17:36:52.675+0000 I CONTROL [initandlisten] distarch: x86_64
      2020-11-22T17:36:52.675+0000 I CONTROL [initandlisten] target_arch: x86_64
      2020-11-22T17:36:52.675+0000 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/usr/pkg/unifi/data/db" } }
      2020-11-22T17:36:52.928+0000 I STORAGE [initandlisten] exception in initAndListen: 98 Unable to lock file: /usr/pkg/unifi/data/db/mongod.lock Resource temporarily unavailable. Is a mongod instance already running?, terminating
      2020-11-22T17:36:52.929+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
      2020-11-22T17:36:52.929+0000 I NETWORK [initandlisten] shutdown: going to flush diaglog...
      2020-11-22T17:36:52.930+0000 I CONTROL [initandlisten] now exiting
      2020-11-22T17:36:52.930+0000 I CONTROL [initandlisten] shutting down with code:100
      (root@iris /usr/pkg/unifi)6% bin/mongod --dbpath /usr/pkg/unifi/data/db --repair
      2020-11-22T17:37:10.309+0000 I CONTROL [initandlisten] MongoDB starting : pid=28748 port=27017 dbpath=/usr/pkg/unifi/data/db 64-bit host=iris.absd.org
      2020-11-22T17:37:10.312+0000 I CONTROL [initandlisten] db version v3.4.24
      2020-11-22T17:37:10.312+0000 I CONTROL [initandlisten] git version: 865b4f6a96d0f5425e39a18337105f33e8db504d
      2020-11-22T17:37:10.312+0000 I CONTROL [initandlisten] allocator: system
      2020-11-22T17:37:10.312+0000 I CONTROL [initandlisten] modules: none
      2020-11-22T17:37:10.313+0000 I CONTROL [initandlisten] build environment:
      2020-11-22T17:37:10.313+0000 I CONTROL [initandlisten] distarch: x86_64
      2020-11-22T17:37:10.313+0000 I CONTROL [initandlisten] target_arch: x86_64
      2020-11-22T17:37:10.314+0000 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/usr/pkg/unifi/data/db" } }
      2020-11-22T17:37:10.315+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
      2020-11-22T17:37:10.315+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=1535M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
      
      2020-11-22T17:44:51.185+0000 E STORAGE [initandlisten] WiredTiger error (0) [1606067091:185070][28748:0x7759113e2800], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 20480B block at offset 212992: block header checksum of 4113582198 doesn't match expected checksum of 104923011
      2020-11-22T17:44:51.185+0000 E STORAGE [initandlisten] WiredTiger error (0) [1606067091:185493][28748:0x7759113e2800], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value
      2020-11-22T17:44:51.185+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1606067091:185814][28748:0x7759113e2800], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2020-11-22T17:44:51.186+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
      2020-11-22T17:44:51.186+0000 I - [initandlisten]
      ***aborting after fassert() failure
      
      2020-11-22T17:44:51.383+0000 F - [initandlisten] Got signal: 6 (Abort trap).
      

       

       

            Assignee:
            eric.sedor@mongodb.com Eric Sedor
            Reporter:
            abs@absd.org David Brownlee
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: