[SERVER-16210] stack trace/error on shutdown in wiredtiger Created: 18/Nov/14 Updated: 10/Dec/14 Resolved: 19/Nov/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Asya Kamsky | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | wiredtiger | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
On shutdown got this in the logs:
Latest pull from master:
|
| Comments |
| Comment by Asya Kamsky [ 27/Nov/14 ] |
|
You must make sure that you don't have any files in the directory that don't belong and that are updated outside of WT's "knowledge". In my case I was shutting down mongod, deleting and recreating the directory but I was doing it without waiting for the full shutdown, so WT files were changed (by me, as it turns out). If you are doing something like this, make sure you use something like killall -w mongod (to wait for it to shut down) before restarting or doing anything to the directory or files. There was no "bug" here - it was interference with DB files outside the process that was causing this. |
| Comment by Dmitriy Selivanov [ 26/Nov/14 ] |
|
I recieve the the same error after restarting server after many updates job (~21M). How can I start it now? Tried mongod --repair, no success. 2014-11-26T16:29:59.515+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends , storage: { dbPath: "/storage/data", engine: "wiredtiger", indexBuildRetry: false, journa , wiredTiger: { engineConfig: "cache_size=26G" }}, systemLog: { destination: "file", logAppend: true, path: "/storage/logs/mongodb.log" } } , {"b":"400000","o":"AE8D61"}, {"b":"400000","o":"ACE041"}, {"b":"400000","o":"95359A"}, {"b":"400000","o":"F65610"}, {"b":"400000","o":"F658D5"}, {"b":"400000","o":"F65D 41"}, {"b":"400000","o":"EC5ABE"}, {"b":"400000","o":"EC5F08"}, {"b":"400000","o":"EC3363"}, {"b":"400000","o":"EC6C06"}, {"b":"400000","o":"EDDE59"}, {"b":"400000","o":"F139FB"}, {"b":"400000","o":"F64C38"}, {"b" :"400000","o":"F64D9A"}, {"b":"400000","o":"F1F95D"}, {"b":"400000","o":"F6397A"}, {"b":"400000","o":"F30463"}, {"b":"400000","o":"F637C1"}, {"b":"400000","o":"F63BE5"}, {"b":"400000","o":"965388"}, {"b":"400000" ,"o":"954708"}, {"b":"400000","o":"952F30"}, {"b":"400000","o":"66A708"}, {"b":"400000","o":"3DA5EC"}, {"b":"400000","o":"3DF304"}, {"b":"7F62C3D77000","o":"21EC5"}, {"b":"400000","o":"3D8259"}],"processInfo":{ , "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFA56E8000", "elfType" : 3 }, { "b" : "7F62C4D69000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elf Type" : 3 }, { "b" : "7F62C4B61000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F62C495D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F62C465900 0", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F62C4353000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F62C413D000", "path" : "/lib/x86_64- linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F62C3D77000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F62C4F87000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }} ***aborting after fassert() failure 2014-11-26T16:30:04.391+0000 F - [initandlisten] Got signal: 6 (Aborted). 0xf41389 0xf40cf2 0xf41076 0x7f62c3dadc30 0x7f62c3dadbb9 0x7f62c3db0fc8 0xece086 0xd5359a 0x1365610 0x13658d5 0x1365d41 0x12c5abe 0x12c5f08 0x12c3363 0x12c6c06 0x12dde59 0x13139fb 0x1364c38 0x1364d9a 0x131f95d 0x136397a 0x1330463 0x13637c1 0x1363be5 0xd65388 0xd54708 0xd52f30 0xa6a708 0x7da5ec 0x7df304 0x7f62c3d98ec5 0x7d8259 , {"b":"400000","o":"B40CF2"}, {"b":"400000","o":"B41076"}, {"b":"7F62C3D77000","o":"36C30"}, {"b":"7F62C3D77000","o":"36BB9"}, {"b":"7F62C3D77000","o":"39FC8"}, {"b":"400000","o":"ACE086"}, {"b":"400000","o":"95359A"}, {"b":"400000","o":"F65610"}, {"b":"400000","o":"F658D5"}, {"b":"400000","o":"F65D41"}, {"b":"400000","o":"EC5ABE"}, {"b":"400000","o":"EC5F08"}, {"b":"400000","o":"EC3363"}, {"b":"400000","o":"EC6C06"}, {"b":"400000","o":"EDDE59"}, {"b":"400000","o":"F139FB"}, {"b":"400000","o":"F64C38"}, {"b":"400000","o":"F64D9A"}, {"b":"400000","o":"F1F95D"}, {"b":"400000","o":"F6397A"}, {"b":"400000","o":"F30463"}, {"b":"400000","o":"F637C1"}, {"b":"400000","o":"F63BE5"}, {"b":"400000","o":"965388"}, {"b":"400000","o":"954708"}, {"b":"400000","o":"952F30"}, {"b":"400000","o":"66A708"}, {"b":"400000","o":"3DA5EC"}, {"b":"400000","o":"3DF304"}, {"b":"7F62C3D77000","o":"21EC5"}, {"b":"400000","o":"3D8259"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc2-pre-", "gitVersion" : "e4329ae5e7bfb56ef9b63a8388285b70b7ec9634", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFA56E8000", "elfType" : 3 }, { "b" : "7F62C4D69000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F62C4B61000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F62C495D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F62C4659000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F62C4353000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F62C413D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F62C3D77000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F62C4F87000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }} |
| Comment by Daniel Pasette (Inactive) [ 19/Nov/14 ] |
|
Closing this as WAD. |
| Comment by Daniel Pasette (Inactive) [ 18/Nov/14 ] |
|
asya, any further information on steps taken to get in this state? |