[SERVER-22811] mongod crashed when starting with wiredTiger files from running server Created: 23/Feb/16  Updated: 02/Mar/16  Resolved: 01/Mar/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Edik Mkoyan Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Copy data directory to another server.
Run mongod with --dbpath flag specifying the copied files.

Participants:

 Description   

I have copied files from data directory from one server to backup the database, when I try to run mongod instance on another server with the backed up files this is what happens.

2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] MongoDB starting : pid=5090 port=27017 dbpath=/root/mongodb 64-bit host=skynet.tumo.lab
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] db version v3.2.3
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] allocator: tcmalloc
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] modules: none
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten] build environment:
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten]     distmod: ubuntu1404
2016-02-23T11:08:30.161+0400 I CONTROL  [initandlisten]     distarch: x86_64
2016-02-23T11:08:30.162+0400 I CONTROL  [initandlisten]     target_arch: x86_64
2016-02-23T11:08:30.162+0400 I CONTROL  [initandlisten] options: { storage: { dbPath: "/root/mongodb" } }
2016-02-23T11:08:30.187+0400 I -        [initandlisten] Detected data files in /root/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-02-23T11:08:30.187+0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-02-23T11:08:30.670+0400 E STORAGE  [initandlisten] WiredTiger (0) [1456211310:670890][5090:0x7f31a4f45d00], file:collection-4--2933107712997422602.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 223674368: block header checksum of 1785676895 doesn't match expected checksum of 278520283
2016-02-23T11:08:30.671+0400 E STORAGE  [initandlisten] WiredTiger (0) [1456211310:671003][5090:0x7f31a4f45d00], file:collection-4--2933107712997422602.wt, WT_SESSION.open_cursor: collection-4--2933107712997422602.wt: encountered an illegal file format or internal value
2016-02-23T11:08:30.671+0400 E STORAGE  [initandlisten] WiredTiger (-31804) [1456211310:671047][5090:0x7f31a4f45d00], file:collection-4--2933107712997422602.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-02-23T11:08:30.671+0400 I -        [initandlisten] Fatal Assertion 28558
2016-02-23T11:08:30.671+0400 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2016-02-23T11:08:30.699+0400 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x12f14b2 0x12f0609 0x12f0e12 0x7f31a38c1340 0x7f31a3522cc9 0x7f31a35260d8 0x127b982 0x107a7b3 0x1a343fc 0x1a3459d 0x1a34984 0x196cc0c 0x196d183 0x196a167 0x196df6e 0x198a738 0x19be700 0x1a331bb 0x1a336d7 0x1a3382b 0x19cc8b8 0x1a309d5 0x19e2763 0x1a30859 0x1a30de9 0x1074fb4 0x1073970 0x106fc52 0x106eb2e 0x1060c2c 0xfc57f4 0xfcaa52 0x105f6e8 0xf8c478 0x992191 0x94d89d 0x7f31a350dec5 0x98fd7c
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EF14B2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EF0609"},{"b":"400000","o":"EF0E12"},{"b":"7F31A38B1000","o":"10340"},{"b":"7F31A34EC000","o":"36CC9","s":"gsignal"},{"b":"7F31A34EC000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E7B982","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C7A7B3"},{"b":"400000","o":"16343FC","s":"__wt_eventv"},{"b":"400000","o":"163459D","s":"__wt_err"},{"b":"400000","o":"1634984","s":"__wt_panic"},{"b":"400000","o":"156CC0C","s":"__wt_block_extlist_read"},{"b":"400000","o":"156D183","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"156A167","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"156DF6E"},{"b":"400000","o":"158A738","s":"__wt_btree_open"},{"b":"400000","o":"15BE700","s":"__wt_conn_btree_open"},{"b":"400000","o":"16331BB","s":"__wt_session_get_btree"},{"b":"400000","o":"16336D7","s":"__wt_session_get_btree"},{"b":"400000","o":"163382B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"15CC8B8","s":"__wt_curfile_open"},{"b":"400000","o":"16309D5"},{"b":"400000","o":"15E2763","s":"__wt_curtable_open"},{"b":"400000","o":"1630859"},{"b":"400000","o":"1630DE9"},{"b":"400000","o":"C74FB4","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C73970","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o":"C6FC52","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"400000","o":"C6EB2E","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"400000","o":"C60C2C","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"400000","o":"BC57F4","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb"},{"b":"400000","o":"BCAA52","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"400000","o":"C5F6E8"},{"b":"400000","o":"B8C478","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"592191","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54D89D","s":"main"},{"b":"7F31A34EC000","o":"21EC5","s":"__libc_start_main"},{"b":"400000","o":"58FD7C"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-77-generic", "version" : "#121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "C1CD0F405485844DA016C6B5275C8BEF3D68DB7A" }, { "b" : "7FFF9D6F7000", "elfType" : 3, "buildId" : "2BB17317A4E4F907D13D05EBB9665AC8184C1E6E" }, { "b" : "7F31A4AD6000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F31A46FB000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F31A44F3000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "B376100CAB1EAC4E5DE066EACFC282BF7C0B54F3" }, { "b" : "7F31A42EF000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "67699FFDA9FD2A552032E0652A242E82D65AA10D" }, { "b" : "7F31A3FEB000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "D0E735DBECD63462DA114BD3F76E6EC7BB1FACCC" }, { "b" : "7F31A3CE5000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "EF3F6DFFA1FBE48436EC6F45CD3AABA157064BB4" }, { "b" : "7F31A3ACF000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F31A38B1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "AF06068681750736E0524DF17D5A86CB2C3F765C" }, { "b" : "7F31A34EC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5382058B69031CAA9B9996C11061CD164C9398FF" }, { "b" : "7F31A4D35000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2A816C3EBBA4E12813FBD34B06FBD25BC892A67F" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12f14b2]
 mongod(+0xEF0609) [0x12f0609]
 mongod(+0xEF0E12) [0x12f0e12]
 libpthread.so.0(+0x10340) [0x7f31a38c1340]
 libc.so.6(gsignal+0x39) [0x7f31a3522cc9]
 libc.so.6(abort+0x148) [0x7f31a35260d8]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x127b982]
 mongod(+0xC7A7B3) [0x107a7b3]
 mongod(__wt_eventv+0x40C) [0x1a343fc]
 mongod(__wt_err+0x8D) [0x1a3459d]
 mongod(__wt_panic+0x24) [0x1a34984]
 mongod(__wt_block_extlist_read+0x6C) [0x196cc0c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x196d183]
 mongod(__wt_block_checkpoint_load+0x3B7) [0x196a167]
 mongod(+0x156DF6E) [0x196df6e]
 mongod(__wt_btree_open+0xC68) [0x198a738]
 mongod(__wt_conn_btree_open+0x140) [0x19be700]
 mongod(__wt_session_get_btree+0xEB) [0x1a331bb]
 mongod(__wt_session_get_btree+0x607) [0x1a336d7]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a3382b]
 mongod(__wt_curfile_open+0x1E8) [0x19cc8b8]
 mongod(+0x16309D5) [0x1a309d5]
 mongod(__wt_curtable_open+0x393) [0x19e2763]
 mongod(+0x1630859) [0x1a30859]
 mongod(+0x1630DE9) [0x1a30de9]
 mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0x94) [0x1074fb4]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x1073970]
 mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x92) [0x106fc52]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x32E) [0x106eb2e]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0xEC) [0x1060c2c]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb+0x204) [0xfc57f4]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x682) [0xfcaa52]
 mongod(+0xC5F6E8) [0x105f6e8]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf8c478]
 mongod(_ZN5mongo13initAndListenEi+0x321) [0x992191]
 mongod(main+0x15D) [0x94d89d]
 libc.so.6(__libc_start_main+0xF5) [0x7f31a350dec5]
 mongod(+0x58FD7C) [0x98fd7c]
-----  END BACKTRACE  -----
Aborted



 Comments   
Comment by Edik Mkoyan [ 02/Mar/16 ]

I thought with journal enabled it will be ok.

Comment by Ramon Fernandez Marina [ 01/Mar/16 ]

edikmkoyan, this backup procedure (copying files off the dbpath of a running mongod) is not supported. Please see recommended backup methods for alternatives.

Regards,
Ramón.

Comment by Edik Mkoyan [ 29/Feb/16 ]

@Thomas,

When the mongod is running on server A, but there is no load on it, I copy the content of dbpath to server B, set the correct permissions and ownership and run the mongod --dbpath /path/to/copy/of/data/directory and see this in my terminal window.

P.S. Apologies for delay.
P.P.S @ tagging is not working here.

Comment by Kelsey Schubert [ 23/Feb/16 ]

Hi edikmkoyan,

Can you please walk me through the steps you took copy the files from the source MongoDB to the target MongoDB (the one displaying this issue on startup)?

In particular, please be sure to answer the following:

  1. Was the source MongoDB server running when you copied the files?
  2. What technique did you use to copy the files? For example, did you take a snapshot or use cp?

Thank you,
Thomas

Comment by Edik Mkoyan [ 23/Feb/16 ]

@Ramon Fernandez f you mean the backup/destination server, the data directory was empty there.
I have created the server from the scratch.

Comment by Ramon Fernandez Marina [ 23/Feb/16 ]

edikmkoyan, was the other server running at the time you copied the files?

Generated at Thu Feb 08 04:01:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.