Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-41173

Can't repair corrupted database from bad shutdown: Segmentation Fault 11

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • Environment:
      Mac 10.13.6 with mongod v4.0.3.

      (edit: Transferring content from here)

      I just ran into a problem in my Node.js script. All the script did was write collection.update(...) and did an upsert of a single record. Nothing fancy. And it throws this error:

      MongoNetworkError: connection 0 to localhost:27017 closed
    at Socket.<anonymous> (./node_modules/mongodb-core/lib/connection/connection.js:352:9)
    at Object.onceWrapper (events.js:285:13)
    at Socket.emit (events.js:197:13)
    at TCP._handle.close (net.js:611:12)
  name: 'MongoNetworkError',
  errorLabels: [ 'TransientTransactionError' ]

      So then I ran mongod --repair:

      $ /usr/local/opt/mongodb30/bin/mongod --repair -v --config /usr/local/etc/mongod.conf --dbpath /Volumes/My\ Passport\ for\ Mac/mongo

      ...which gave this output (the final result of tail -f /usr/local/var/log/mongodb/mongo.log).

      2019-05-15T20:48:38.470-0700 D INDEX    [initandlisten]    bulk commit starting for index: _id_
2019-05-15T20:48:41.429-0700 D STORAGE  [initandlisten] allocating new extent
2019-05-15T20:48:41.552-0700 D STORAGE  [initandlisten] MmapV1ExtentManager::allocateExtent desiredSize:524288 fromFreeList: 0 eloc: 10:3a279000
2019-05-15T20:48:41.597-0700 D STORAGE  [initandlisten] example.papers: clearing plan cache - collection info cache reset
2019-05-15T20:48:42.436-0700 D STORAGE  [initandlisten] example.papers: clearing plan cache - collection info cache reset
2019-05-15T20:48:42.504-0700 D STORAGE  [initandlisten] example.papers: clearing plan cache - collection info cache reset
2019-05-15T20:48:42.970-0700 I STORAGE  [initandlisten] cleaning up failed repair db: example path: /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_0
2019-05-15T20:48:43.885-0700 I -        [initandlisten] Fatal assertion 18506 DuplicateKey E11000 duplicate key error index: example.papers.$_id_ dup key: { : ObjectId('5cdbf188258c9997819e553c') }
2019-05-15T20:48:43.906-0700 I CONTROL  [initandlisten]
 0x10cef4c2a 0x10cea195e 0x10ce8e937 0x10c7668fa 0x10c7639cc 0x10c762643 0x10c7660ac 0x7fff52856015 0x7
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"10C761000","o":"793C2A","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"10C761000","o":"74095E","s":"_ZN5mongo10logContextEPKc"},{"b":"10C761000","o":"72D937","s":"_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE"},{"b":"10C761000","o":"58FA","s":"_ZN5mongoL30repairDatabasesAndCheckVersionEv"},{"b":"10C761000","o":"29CC","s":"_ZN5mongoL14_initAndListenEi"},{"b":"10C761000","o":"1643","s":"_ZN5mongo13initAndListenEi"},{"b":"10C761000","o":"50AC","s":"main"},{"b":"7FFF52855000","o":"1015","s":"start"},{"b":"0","o":"7"}],"processInfo":{ "mongodbVersion" : "3.0.12", "gitVersion" : "33934938e0e95d534cebbaff656cde916b9c3573", "uname" : { "sysname" : "Darwin", "release" : "17.7.0", "version" : "Darwin Kernel Version 17.7.0: Fri Nov  2 20:43:16 PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/usr/local/opt/mongodb30/bin/mongod", "machType" : 2, "b" : "10C761000", "vmaddr" : "100000000", "buildId" : "239CC35B9A1A3C7AB990330C26274C60" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF5056F000", "vmaddr" : "7FFF4D7EC000", "buildId" : "9A40C6279ADE361789E70733BDBE4E27" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF507A4000", "vmaddr" : "7FFF4DA21000", "buildId" : "7D3DACCC3804393CABC11A580FD00CB6" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF526EC000", "vmaddr" : "7FFF4F969000", "buildId" : "092479CB10083A83BECFE115F24D13C1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF526F1000", "vmaddr" : "7FFF4F96E000", "buildId" : "029F59859B6E3DCB9B96FD007678C6A7" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF526FC000", "vmaddr" : "7FFF4F979000", "buildId" : "968B8E3F368132309D78BB8732024F6E" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF52704000", "vmaddr" : "7FFF4F981000", "buildId" : "3885083D50D83EECB481B2E605180D7F" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF5270E000", "vmaddr" : "7FFF4F98B000", "buildId" : "495BACA267D7369DABB1FE67FAF63A6A" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF5281B000", "vmaddr" : "7FFF4FA98000", "buildId" : "BF3685492DFB3530B4CB31D5EDAC4F2F" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF52855000", "vmaddr" : "7FFF4FAD2000", "buildId" : "49BF9E96829730CF8AA6128CC14054B2" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF52873000", "vmaddr" : "7FFF4FAF0000", "buildId" : "E34E283E90FA3C59B48E1277CDB9CDCE" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF52881000", "vmaddr" : "7FFF4FAFE000", "buildId" : "B184B521FF243142AFAF23D170CF918C" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF52882000", "vmaddr" : "7FFF4FAFF000", "buildId" : "1902A611081A3452B11EEBD1B166E831" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF52887000", "vmaddr" : "7FFF4FB04000", "buildId" : "26C0BA228F933A079A4EC8D53D2CE42E" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF5288A000", "vmaddr" : "7FFF4FB07000", "buildId" : "711E18B25BBE3211A91656740C27D17A" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF5288C000", "vmaddr" : "7FFF4FB09000", "buildId" : "39E46A6FB2283E78B83E1779F9707A39" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF528A4000", "vmaddr" : "7FFF4FB21000", "buildId" : "17303FDF0D2D3963B05EB4DF63052D47" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF528A5000", "vmaddr" : "7FFF4FB22000", "buildId" : "1187BFE8457632478177481554E1F9E7" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF5292F000", "vmaddr" : "7FFF4FBAC000", "buildId" : "DF6B5287203E30CB994778DF446C72B8" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF52933000", "vmaddr" : "7FFF4FBB0000", "buildId" : "486000D3D8CB3BE78EE58BF380DE6DF7" }, { "path" : "/usr/lib/system/libsystem_darwin.dylib", "machType" : 6, "b" : "7FFF52937000", "vmaddr" : "7FFF4FBB4000", "buildId" : "09C21A4A9EE0388BA9D9DFF8F6758791" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF52939000", "vmaddr" : "7FFF4FBB6000", "buildId" : "976168E40F953725800FCEC6166C9E76" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF52940000", "vmaddr" : "7FFF4FBBD000", "buildId" : "AB634A98B8AA3804843638261FC8EC4D" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF529B1000", "vmaddr" : "7FFF4FC2E000", "buildId" : "17570F46566C39FCBEF6635A355DD549" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF529FD000", "vmaddr" : "7FFF4FC7A000", "buildId" : "7FD437359DDD300E8C4AF909A74BDF49" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF52A1D000", "vmaddr" : "7FFF4FC9A000", "buildId" : "5E86B2DE9E15335487144094ED5F698D" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF52B4E000", "vmaddr" : "7FFF4FDCB000", "buildId" : "9DC03712552D3AEE9519B5ED70980B70" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF52B59000", "vmaddr" : "7FFF4FDD6000", "buildId" : "08012EC02CD234BEBF93E7F56491299A" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF52B77000", "vmaddr" : "7FFF4FDF4000", "buildId" : "553DFCC69D313B9CAB7C30F6F265786D" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF52B7B000", "vmaddr" : "7FFF4FDF8000", "buildId" : "DE8D14E8A2763FF8AE1377F7040F33C1" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF5298A000", "vmaddr" : "7FFF4FC07000", "buildId" : "0B12D320E66C3DCE9713C7B8D03C97D7" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF52B63000", "vmaddr" : "7FFF4FDE0000", "buildId" : "6355EE2D54563CA8A227B96E8F1E2AF8" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF52B6B000", "vmaddr" : "7FFF4FDE8000", "buildId" : "0E51CCBA91F234E1BF2AFEEFD3D321E4" }, { "path" : "/usr/lib/system/libsystem_symptoms.dylib", "machType" : 6, "b" : "7FFF52B7D000", "vmaddr" : "7FFF4FDFA000", "buildId" : "441C6CA057113BB18420DDAC3D5272E1" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF52B85000", "vmaddr" : "7FFF4FE02000", "buildId" : "3A6CB7068CA636168AFC14AAD7FAF187" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF52B9A000", "vmaddr" : "7FFF4FE17000", "buildId" : "BEF3FB4956043B5F82B5332B80023AC3" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF52BA0000", "vmaddr" : "7FFF4FE1D000", "buildId" : "0BC7AD67671D31D48B88C317B8379598" }, { "path" : "/usr/lib/closure/libclosured.dylib", "machType" : 6, "b" : "7FFF50035000", "vmaddr" : "7FFF4D2B2000", "buildId" : "112BC241662638488DD8B34B5B6F7ABC" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF51C28000", "vmaddr" : "7FFF4EEA5000", "buildId" : "DD9E5EC5B5073249B70093433E2D5EDF" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF507FB000", "vmaddr" : "7FFF4DA78000", "buildId" : "EF5E37D711D93530BE45B986612D13E2" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x3A) [0x10cef4c2a]
 mongod(_ZN5mongo10logContextEPKc+0x10E) [0x10cea195e]
 mongod(_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE+0x107) [0x10ce8e937]
 mongod(_ZN5mongoL30repairDatabasesAndCheckVersionEv+0x35A) [0x10c7668fa]
 mongod(_ZN5mongoL14_initAndListenEi+0xE9C) [0x10c7639cc]
 mongod(_ZN5mongo13initAndListenEi+0x13) [0x10c762643]
 mongod(main+0x40C) [0x10c7660ac]
 libdyld.dylib(start+0x1) [0x7fff52856015]
 ??? [0x7]
-----  END BACKTRACE  -----
2019-05-15T20:48:43.906-0700 I -        [initandlisten]

***aborting after fassert() failure

      It failed :/. So I tried restarting the mongod process, and got this in the logs:

      2019-05-15T20:52:28.086-0700 I CONTROL  ***** SERVER RESTARTED *****
2019-05-15T20:52:28.115-0700 D SHARDING isInRangeTest passed
2019-05-15T20:52:28.115-0700 I CONTROL  [initandlisten] MongoDB starting : pid=6589 port=27017 dbpath=/Volumes/My Passport for Mac/mongo 64-bit host=Lances-MacBook-Pro.local
2019-05-15T20:52:28.115-0700 I CONTROL  [initandlisten] db version v3.0.12
2019-05-15T20:52:28.115-0700 I CONTROL  [initandlisten] git version: 33934938e0e95d534cebbaff656cde916b9c3573
2019-05-15T20:52:28.115-0700 I CONTROL  [initandlisten] build info: Darwin yosemitevm.local 14.5.0 Darwin Kernel Version 14.5.0: Thu Apr 21 20:40:54 PDT 2016; root:xnu-2782.50.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
2019-05-15T20:52:28.115-0700 I CONTROL  [initandlisten] allocator: system
2019-05-15T20:52:28.115-0700 I CONTROL  [initandlisten] options: { config: "/usr/local/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/Volumes/My Passport for Mac/mongo" }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/var/log/mongodb/mongo.log", verbosity: 1 } }
2019-05-15T20:52:28.115-0700 D NETWORK  [initandlisten] fd limit hard:9223372036854775807 soft:4864 max conn: 3891
2019-05-15T20:52:28.209-0700 W -        [initandlisten] Detected unclean shutdown - /Volumes/My Passport for Mac/mongo/mongod.lock is not empty.
2019-05-15T20:52:28.224-0700 I STORAGE  [initandlisten] **************
old lock file: /Volumes/My Passport for Mac/mongo/mongod.lock.  probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
please make sure that your journal directory is mounted.
found 2 dbs.
see: http://dochub.mongodb.org/core/repair for more information
*************
2019-05-15T20:52:28.224-0700 D -        [initandlisten] User Assertion: 12596:old lock file
2019-05-15T20:52:28.230-0700 I STORAGE  [initandlisten] exception in initAndListen: 12596 old lock file, terminating
2019-05-15T20:52:28.231-0700 I CONTROL  [initandlisten] dbexit:  rc: 100

      Wondering what I can do given I have no backups. I am not very far along but I would like to not have to rebuild the database. Wondering if I can recover anything and get it to restart again. I'm also not really sure what exactly I did to cause this.

      This is related to the error.

      $ /usr/local/opt/mongodb30/bin/mongod --version
db version v3.0.12

      Now, I upgraded to > 4.0, and ran start and repair, which ends up just seg faulting:

      $ mongod -v --config /usr/local/etc/mongod.conf --dbpath /Volumes/My\ Passport\ for\ Mac/mongo
2019-05-15T21:28:12.894-0700 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
Error 100
$ mongod --repair -v --config /usr/local/etc/mongod.conf --dbpath /Volumes/My\ Passport\ for\ Mac/mongo
2019-05-15T21:28:46.557-0700 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
Segmentation fault: 11
$ mongod --repair --config /usr/local/etc/mongod.conf --dbpath /Volumes/My\ Passport\ for\ Mac/mongo
2019-05-15T21:29:00.268-0700 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
Segmentation fault: 11

      The logs from that show:

      2019-05-15T21:35:04.761-0700 I STORAGE  [initandlisten] repairDatabase local
2019-05-15T21:35:05.247-0700 I STORAGE  [initandlisten] allocating new ns file /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.ns, filling with zeroes...
2019-05-15T21:35:05.565-0700 I STORAGE  [initandlisten] done allocating ns file /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.ns, size: 16MB, took 0.317 seconds
2019-05-15T21:35:05.565-0700 I STORAGE  [FileAllocator] allocating new datafile /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.0
2019-05-15T21:35:05.565-0700 I STORAGE  [FileAllocator] creating directory /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/_tmp
2019-05-15T21:35:06.425-0700 I STORAGE  [FileAllocator] done allocating datafile /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.0, size: 64MB,  took 0.859 secs
2019-05-15T21:35:06.771-0700 W ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
2019-05-15T21:35:06.783-0700 F -        [initandlisten] Invalid access at address: 0x0
2019-05-15T21:35:06.801-0700 F -        [initandlisten] Got signal: 11 (Segmentation fault: 11).
      Original description

      I have written up the detailed description here

      Essentially what I'm getting now is this:

      {{}}

      {{$ mongod -v --config /usr/local/etc/mongod.conf --dbpath /Volumes/My\ Passport\ for\ Mac/mongo 2019-05-15T21:28:12.894-0700 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none' Error 100 }}
      {{$ mongod --repair -v --config /usr/local/etc/mongod.conf --dbpath /Volumes/My\ Passport\ for\ Mac/mongo 2019-05-15T21:28:46.557-0700 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none' Segmentation fault: 11 }}
      $ mongod --repair --config /usr/local/etc/mongod.conf --dbpath /Volumes/My\ Passport\ for\ Mac/mongo 2019-05-15T21:29:00.268-0700 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none' Segmentation fault: 11

      And the logs show:

      2019-05-15T21:35:04.761-0700 I STORAGE  [initandlisten] repairDatabase local
2019-05-15T21:35:05.247-0700 I STORAGE  [initandlisten] allocating new ns file /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.ns, filling with zeroes...
2019-05-15T21:35:05.565-0700 I STORAGE  [initandlisten] done allocating ns file /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.ns, size: 16MB, took 0.317 seconds
2019-05-15T21:35:05.565-0700 I STORAGE  [FileAllocator] allocating new datafile /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.0
2019-05-15T21:35:05.565-0700 I STORAGE  [FileAllocator] creating directory /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/_tmp
2019-05-15T21:35:06.425-0700 I STORAGE  [FileAllocator] done allocating datafile /Volumes/My Passport for Mac/mongo/_tmp_repairDatabase_2/local.0, size: 64MB,  took 0.859 secs
2019-05-15T21:35:06.771-0700 W ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
2019-05-15T21:35:06.783-0700 F -        [initandlisten] Invalid access at address: 0x0
2019-05-15T21:35:06.801-0700 F -        [initandlisten] Got signal: 11 (Segmentation fault: 11).

            Assignee:
            eric.sedor@mongodb.com Eric Sedor
            Reporter:
            mawati Lokasa Mawati
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: