Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22811

mongod crashed when starting with wiredTiger files from running server

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.3
    • Component/s: WiredTiger
    • None
    • ALL
    • Hide

      Copy data directory to another server.
      Run mongod with --dbpath flag specifying the copied files.

      Show
      Copy data directory to another server. Run mongod with --dbpath flag specifying the copied files.
    • None
    • 0
    • None
    • None
    • None
    • None
    • None
    • None

      I have copied files from data directory from one server to backup the database, when I try to run mongod instance on another server with the backed up files this is what happens.

      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] MongoDB starting : pid=5090 port=27017 dbpath=/root/mongodb 64-bit host=skynet.tumo.lab
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] db version v3.2.3
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] allocator: tcmalloc
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] modules: none
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] build environment:
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten]     distmod: ubuntu1404
      2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten]     distarch: x86_64
      2016-02-23T11:08:30.162+0400 I CONTROL  [initandlisten]     target_arch: x86_64
      2016-02-23T11:08:30.162+0400 I CONTROL  [initandlisten] options: { storage: { dbPath: "/root/mongodb" } }
      2016-02-23T11:08:30.187+0400 I -        [initandlisten] Detected data files in /root/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2016-02-23T11:08:30.187+0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2016-02-23T11:08:30.670+0400 E STORAGE  [initandlisten] WiredTiger (0) [1456211310:670890][5090:0x7f31a4f45d00], file:collection-4--2933107712997422602.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 223674368: block header checksum of 1785676895 doesn't match expected checksum of 278520283
      2016-02-23T11:08:30.671+0400 E STORAGE  [initandlisten] WiredTiger (0) [1456211310:671003][5090:0x7f31a4f45d00], file:collection-4--2933107712997422602.wt, WT_SESSION.open_cursor: collection-4--2933107712997422602.wt: encountered an illegal file format or internal value
      2016-02-23T11:08:30.671+0400 E STORAGE  [initandlisten] WiredTiger (-31804) [1456211310:671047][5090:0x7f31a4f45d00], file:collection-4--2933107712997422602.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2016-02-23T11:08:30.671+0400 I -        [initandlisten] Fatal Assertion 28558
      2016-02-23T11:08:30.671+0400 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2016-02-23T11:08:30.699+0400 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x12f14b2 0x12f0609 0x12f0e12 0x7f31a38c1340 0x7f31a3522cc9 0x7f31a35260d8 0x127b982 0x107a7b3 0x1a343fc 0x1a3459d 0x1a34984 0x196cc0c 0x196d183 0x196a167 0x196df6e 0x198a738 0x19be700 0x1a331bb 0x1a336d7 0x1a3382b 0x19cc8b8 0x1a309d5 0x19e2763 0x1a30859 0x1a30de9 0x1074fb4 0x1073970 0x106fc52 0x106eb2e 0x1060c2c 0xfc57f4 0xfcaa52 0x105f6e8 0xf8c478 0x992191 0x94d89d 0x7f31a350dec5 0x98fd7c
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"EF14B2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EF0609"},{"b":"400000","o":"EF0E12"},{"b":"7F31A38B1000","o":"10340"},{"b":"7F31A34EC000","o":"36CC9","s":"gsignal"},{"b":"7F31A34EC000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E7B982","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C7A7B3"},{"b":"400000","o":"16343FC","s":"__wt_eventv"},{"b":"400000","o":"163459D","s":"__wt_err"},{"b":"400000","o":"1634984","s":"__wt_panic"},{"b":"400000","o":"156CC0C","s":"__wt_block_extlist_read"},{"b":"400000","o":"156D183","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"156A167","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"156DF6E"},{"b":"400000","o":"158A738","s":"__wt_btree_open"},{"b":"400000","o":"15BE700","s":"__wt_conn_btree_open"},{"b":"400000","o":"16331BB","s":"__wt_session_get_btree"},{"b":"400000","o":"16336D7","s":"__wt_session_get_btree"},{"b":"400000","o":"163382B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"15CC8B8","s":"__wt_curfile_open"},{"b":"400000","o":"16309D5"},{"b":"400000","o":"15E2763","s":"__wt_curtable_open"},{"b":"400000","o":"1630859"},{"b":"400000","o":"1630DE9"},{"b":"400000","o":"C74FB4","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C73970","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o":"C6FC52","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"400000","o":"C6EB2E","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"400000","o":"C60C2C","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"400000","o":"BC57F4","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb"},{"b":"400000","o":"BCAA52","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"400000","o":"C5F6E8"},{"b":"400000","o":"B8C478","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"592191","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54D89D","s":"main"},{"b":"7F31A34EC000","o":"21EC5","s":"__libc_start_main"},{"b":"400000","o":"58FD7C"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-77-generic", "version" : "#121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "C1CD0F405485844DA016C6B5275C8BEF3D68DB7A" }, { "b" : "7FFF9D6F7000", "elfType" : 3, "buildId" : "2BB17317A4E4F907D13D05EBB9665AC8184C1E6E" }, { "b" : "7F31A4AD6000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F31A46FB000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F31A44F3000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "B376100CAB1EAC4E5DE066EACFC282BF7C0B54F3" }, { "b" : "7F31A42EF000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "67699FFDA9FD2A552032E0652A242E82D65AA10D" }, { "b" : "7F31A3FEB000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "D0E735DBECD63462DA114BD3F76E6EC7BB1FACCC" }, { "b" : "7F31A3CE5000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "EF3F6DFFA1FBE48436EC6F45CD3AABA157064BB4" }, { "b" : "7F31A3ACF000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F31A38B1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "AF06068681750736E0524DF17D5A86CB2C3F765C" }, { "b" : "7F31A34EC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5382058B69031CAA9B9996C11061CD164C9398FF" }, { "b" : "7F31A4D35000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2A816C3EBBA4E12813FBD34B06FBD25BC892A67F" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12f14b2]
       mongod(+0xEF0609) [0x12f0609]
       mongod(+0xEF0E12) [0x12f0e12]
       libpthread.so.0(+0x10340) [0x7f31a38c1340]
       libc.so.6(gsignal+0x39) [0x7f31a3522cc9]
       libc.so.6(abort+0x148) [0x7f31a35260d8]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x127b982]
       mongod(+0xC7A7B3) [0x107a7b3]
       mongod(__wt_eventv+0x40C) [0x1a343fc]
       mongod(__wt_err+0x8D) [0x1a3459d]
       mongod(__wt_panic+0x24) [0x1a34984]
       mongod(__wt_block_extlist_read+0x6C) [0x196cc0c]
       mongod(__wt_block_extlist_read_avail+0x33) [0x196d183]
       mongod(__wt_block_checkpoint_load+0x3B7) [0x196a167]
       mongod(+0x156DF6E) [0x196df6e]
       mongod(__wt_btree_open+0xC68) [0x198a738]
       mongod(__wt_conn_btree_open+0x140) [0x19be700]
       mongod(__wt_session_get_btree+0xEB) [0x1a331bb]
       mongod(__wt_session_get_btree+0x607) [0x1a336d7]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a3382b]
       mongod(__wt_curfile_open+0x1E8) [0x19cc8b8]
       mongod(+0x16309D5) [0x1a309d5]
       mongod(__wt_curtable_open+0x393) [0x19e2763]
       mongod(+0x1630859) [0x1a30859]
       mongod(+0x1630DE9) [0x1a30de9]
       mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0x94) [0x1074fb4]
       mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x1073970]
       mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x92) [0x106fc52]
       mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x32E) [0x106eb2e]
       mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0xEC) [0x1060c2c]
       mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb+0x204) [0xfc57f4]
       mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x682) [0xfcaa52]
       mongod(+0xC5F6E8) [0x105f6e8]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf8c478]
       mongod(_ZN5mongo13initAndListenEi+0x321) [0x992191]
       mongod(main+0x15D) [0x94d89d]
       libc.so.6(__libc_start_main+0xF5) [0x7f31a350dec5]
       mongod(+0x58FD7C) [0x98fd7c]
      -----  END BACKTRACE  -----
      Aborted
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            edikmkoyan Edik Mkoyan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: