Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-41791

MongoDb didnt start because of deleted folder related to wiredTiger

    • Type: Icon: Question Question
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.7
    • Component/s: None
    • Labels:
      None
    • Environment:
      ongoDB shell version v3.6.7

      Linux qa-bk-mongo.tloapps.com 4.14.67-66.56.amzn1.x86_64 #1 SMP Tue Sep 4 22:03:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
    • Server Triage

      Hi We are accidentally deleted the folder related to wireTiger.

       

      Please see the output log below during the mongo restart or mongo start

      ----- END BACKTRACE -----
      2019-06-17T00:23:19.644+0000 I CONTROL [main] ***** SERVER RESTARTED *****
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] MongoDB starting : pid=123733 port=27017 dbpath=/mongo/mongodb 64-bit host=qa-bk-mongo.tloapps.com
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] db version v3.6.7
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] git version: 2628472127e9f1826e02c665c1d93880a204075e
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] allocator: tcmalloc
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] modules: enterprise
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] build environment:
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] distmod: amzn64
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] distarch: x86_64
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] target_arch: x86_64
      2019-06-17T00:23:19.653+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

      { bindIp: "10.40.11.46,127.0.0.1", port: 27017 }

      , processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, setParameter: { failIndexKeyTooLong: "false" }, storage: { dbPath: "/mongo/mongodb", directoryPerDB: true, journal:

      { enabled: false }

      , mmapv1: { nsSize: 2047 }, wiredTiger: { engineConfig:

      { directoryForIndexes: true }

      } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", verbosity: 0 } }
      2019-06-17T00:23:19.654+0000 I - [initandlisten] Detected data files in /mongo/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2019-06-17T00:23:19.654+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=60957M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),,log=(enabled=false),
      2019-06-17T00:23:20.313+0000 I STORAGE [initandlisten] WiredTiger message [1560731000:313485][123733:0x7fe667a83b00], txn-recover: Set global recovery timestamp: 0
      2019-06-17T00:23:22.010+0000 E STORAGE [initandlisten] WiredTiger error (2) [1560731002:10411][123733:0x7fe667a83b00], file:reporting_Q1D1_2019/collection/190--608319021127610105.wt, WT_SESSION.open_cursor: /mongo/mongodb/reporting_Q1D1_2019/collection/190--608319021127610105.wt: handle-open: open: No such file or directory
      2019-06-17T00:23:22.010+0000 E STORAGE [initandlisten] no cursor for uri: table:reporting_Q1D1_2019/collection/190--608319021127610105
      2019-06-17T00:23:22.010+0000 F - [initandlisten] Invalid access at address: 0x58
      2019-06-17T00:23:22.021+0000 F - [initandlisten] Got signal: 11 (Segmentation fault).

      0x55f283f4ac11 0x55f283f49e29 0x55f283f4a496 0x7fe664d0f690 0x55f282a50240 0x55f282a5a2a5 0x55f282a43f5f 0x55f2827210de 0x55f28271b816 0x55f2827137d6 0x55f282743217 0x55f2824f27dc 0x55f2825da8fc 0x55f2825610c9 0x7fe664955445 0x55f2825ca301
      ----- BEGIN BACKTRACE -----

      {"backtrace":[\{"b":"55F281B98000","o":"23B2C11","s":"_ZN5mongo15printStackTraceERSo"}

      ,{"b":"55F281B98000","o":"23B1E29"},{"b":"55F281B98000","o":"23B2496"},{"b":"7FE664D00000","o":"F690"},{"b":"55F281B98000","o":"EB8240","s":"ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv"},{"b":"55F281B98000","o":"EC22A5","s":"_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE"},{"b":"55F281B98000","o":"EABF5F","s":"_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE"},{"b":"55F281B98000","o":"B890DE","s":"_ZN5mongo26KVDatabaseCatalogEntryBase14initCollectionEPNS_16OperationContextERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb"},{"b":"55F281B98000","o":"B83816","s":"_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE"},{"b":"55F281B98000","o":"B7B7D6"},{"b":"55F281B98000","o":"BAB217","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55F281B98000","o":"95A7DC"},{"b":"55F281B98000","o":"A428FC","s":"_ZN5mongo11mongoDbMainEiPPcS1"},{"b":"55F281B98000","o":"9C90C9","s":"main"},{"b":"7FE664933000","o":"22445","s":"__libc_start_main"},{"b":"55F281B98000","o":"A32301"}],"processInfo":{ "mongodbVersion" : "3.6.7", "gitVersion" : "2628472127e9f1826e02c665c1d93880a204075e", "compiledModules" : [ "enterprise" ], "uname" :

      { "sysname" : "Linux", "release" : "4.14.67-66.56.amzn1.x86_64", "version" : "#1 SMP Tue Sep 4 22:03:21 UTC 2018", "machine" : "x86_64" }

      , "somap" : [ { "b" : "55F281B98000", "elfType" : 3, "buildId" : "7031182F5479D929C91724A4C6B20B1B756BEF60" }, { "b" : "7FFED6D1B000", "elfType" : 3, "buildId" : "5AD47F0C2A5C2B6F33C2712F9E30AE3B725C7C6F" }, { "b" : "7FE667602000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "8B3C5399A42E45A96F1F2A0B24F60B960B83A453" }, { "b" : "7FE6673B6000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "60A0E0B977CA36916F8141FAF3127C8A2FEF6C5E" }, { "b" : "7FE667190000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "10636588A309CD05273443A77674FC764340AA73" }, { "b" : "7FE666CCC000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "A492A8F56E18651280780000E2948C5832B3BE2B" }, { "b" : "7FE6669F1000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "533DF1E1848F9A43A960558E351EF92B2CFA3B34" }, { "b" : "7FE6667D6000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "354560FFC93703E5A80EEC8C66DF9E59DA335001" }, { "b" : "7FE666584000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "97F36EE026428345EEB18EE7F9EFB048ADB415A7" }, { "b" : "7FE666375000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "BBE520FB0B4F67D5F708D233C31E7047B759068D" }, { "b" : "7FE666128000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B4C91D3D76D4819DDD1F7D2360F600AC13280628" }, { "b" : "7FE665F0F000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "9E5E0BF5F22DE7555BC4B9853240817147489258" }, { "b" : "7FE665AB1000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "0B7F7487280FE68AF9302A282FAE37776A99BC80" }, { "b" : "7FE665840000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "9C4EB34A346260F2A77746F4E5ED837619137DB7" }, { "b" : "7FE66563C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "0E5CD5BAA5EE8BF3648A5031B088F9A78C89364F" }, { "b" : "7FE665434000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "F2701E2A24459D5B55DF5549D585F091E7BCF07A" }, { "b" : "7FE665132000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "07FB92AFEF1756F093371CE60C3AE85DD3A06325" }, { "b" : "7FE664F1C000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "43B97F743BEE26B9F2A3F2272776DC6EC16C8903" }, { "b" : "7FE664D00000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D973C39D1900DC61D8519C653C3BC405692DE563" }, { "b" : "7FE664933000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "AF310F56618FC1EF9158973484F60942F11CC0FB" }, { "b" : "7FE667881000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "8402047FD4A85B3CD1142346EA06BCD6E15A82A3" }, { "b" : "7FE66470E000", "path" : "/usr/lib64/libnghttp2.so.14", "elfType" : 3, "buildId" : "3EE1695029B9ACFFA177E8C98B50FBB5BB3F102C" }, { "b" : "7FE6644ED000", "path" : "/usr/lib64/libidn2.so.0", "elfType" : 3, "buildId" : "5235BD50D3FB450683328735B730532020DEE4BF" }, { "b" : "7FE6642C5000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "64DAC35781655CB55CEF517A261F0DD4EC98C4F5" }, { "b" : "7FE664050000", "path" : "/usr/lib64/libpsl.so.0", "elfType" : 3, "buildId" : "09BFE69665CFEEC18F81D8C4A971DCA29310186C" }, { "b" : "7FE663E05000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "29E348014EF0472AF76744AA690B19CA5B53A567" }, { "b" : "7FE663BDE000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "5164FC4C74515A854F593752354CFBFBA79A237C" }, { "b" : "7FE6638BB000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "F8EBE9C4F06D0F67A3C50B167CE144FAC37E85A9" }, { "b" : "7FE66368E000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "C42137E45CF3EA8FB4DEF9F8ED45EB3A0AC097A7" }, { "b" : "7FE66348A000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "CCD5CF7100E85E680226237177E6D190026EBB0B" }, { "b" : "7FE663285000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "D2470734E74017091271F7CCFA8D3D7E7251DD1C" }, { "b" : "7FE663048000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "FB0BEE2C4D6C18CBDF378525F88E043493D8B3BA" }, { "b" : "7FE662D60000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "295B8EDE0A3D565218B9086DECC047FB28F235E5" }, { "b" : "7FE662B2D000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "68572D6C73CAEE5B0F36E50873B092275DE5697C" }, { "b" : "7FE66292A000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7FE662714000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7FE662509000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "24C5C48F63275B5A1AD2432680A0FAA88E4EF2FB" }, { "b" : "7FE66217F000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "C4EA4E0B2A8552F52928472F0627DEA815DF4579" }, { "b" : "7FE661F65000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "AF2BB3CD9FCDA0B820C88AAB5D80639CA5287921" }, { "b" : "7FE661D2E000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "9A9BE1D2234D3641B9CFE17BB7A40B82973A2802" }, { "b" : "7FE661B2B000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "C74958577D2A3C4D095C32F09E7189BAD67D7EAA" }, { "b" : "7FE66191D000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "2B9C3C8ED0AC9CE675617BDB4174D5EACF1FB0FF" }, { "b" : "7FE66171A000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7FE661404000", "path" : "/usr/lib64/libunistring.so.0", "elfType" : 3, "buildId" : "2B090A6860553944846E3C227B6AD12F279B304F" }, { "b" : "7FE66108E000", "path" : "/usr/lib64/libicuuc.so.50", "elfType" : 3, "buildId" : "06AB750458E6948B6F40F05E705996DB44ADDF9B" }, { "b" : "7FE660E8C000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "5BF9324A39BBE15AFDE1FF221A5121C38DC94A0B" }, { "b" : "7FE660C6B000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" }, { "b" : "7FE65F698000", "path" : "/usr/lib64/libicudata.so.50", "elfType" : 3, "buildId" : "291EDB545286F945CDE3AF6F5CF24FA2F53FDDDA" }, { "b" : "7FE65F394000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "0F007CD9B5110F4ED296D5EC8F28C18F8866899D" } ] }}
      mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55f283f4ac11]
      mongod(+0x23B1E29) [0x55f283f49e29]
      mongod(+0x23B2496) [0x55f283f4a496]
      libpthread.so.0(+0xF690) [0x7fe664d0f690]
      mongod(_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv+0x280) [0x55f282a50240]
      mongod(_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE+0x55) [0x55f282a5a2a5]
      mongod(_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE+0x27F) [0x55f282a43f5f]
      mongod(ZN5mongo26KVDatabaseCatalogEntryBase14initCollectionEPNS_16OperationContextERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x54E) [0x55f2827210de]
      mongod(_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE+0xA06) [0x55f28271b816]
      mongod(+0xB7B7D6) [0x55f2827137d6]
      mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x637) [0x55f282743217]
      mongod(+0x95A7DC) [0x55f2824f27dc]
      mongod(ZN5mongo11mongoDbMainEiPPcS1+0x86C) [0x55f2825da8fc]
      mongod(main+0x9) [0x55f2825610c9]
      libc.so.6(__libc_start_main+0xF5) [0x7fe664955445]
      mongod(+0xA32301) [0x55f2825ca301]
      ----- END BACKTRACE -----

       

       

       

       

      Our problem is, How to fix this issue and so that we may able start the mongo without this problem.

       

       

      Thank you.

            Assignee:
            backlog-server-triage [HELP ONLY] Backlog - Triage Team
            Reporter:
            melvin.bautista@salesify.com Melvin Bautista
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: