[SERVER-16021] WT failed to start with "lsm-worker: Error in LSM worker thread 2: No such file or directory" Created: 21/Oct/14  Updated: 31/Aug/15  Resolved: 23/Jul/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: 3.1.7

Type: Bug Priority: Major - P3
Reporter: Rui Zhang (Inactive) Assignee: Alexander Gorrod
Resolution: Done Votes: 0
Labels: 28qa, 32powercycle, wttt
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

mongod: d0f6dd1ebc338dc96a672057289c5fef56ecad0b
wt: 525b9969876069fa4669cebe11f2ee89368f907b
dataset: as document in CAP-1595


Issue Links:
Depends
is depended on by WT-1973 MongoDB changes for WiredTiger 2.7.0 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

during test, my mongod with WT failed to start.

Steps to restart mongod

#!/bin/bash
 
set -x
 
killall mongod
sleep 5
 
DATADISK=/data/ssd-sdc
DBDIR=$DATADISK/dbs
LOGDIR=$DATADISK/logs/mongod.log
 
date
LD_LIBRARY_PATH=$WT_INSTALL/lib numactl --physcpubind=0-3 --membind=0 -- ./bin/mongod --dbpath=$DBDIR --storageEngine=wiredtiger --logpath=$LOGDIR --fork
date

this setup works fine until it cannot start.



 Comments   
Comment by Jonathan Abrahams [ 31/Jul/15 ]

I found a different issue when running 3.1.7 with LSM: SERVER-19692

Comment by Michael Cahill (Inactive) [ 31/Jul/15 ]

jonathan.abrahams, does this repro on stock 3.1.7 or master?

Comment by Jonathan Abrahams [ 30/Jul/15 ]

I ran the LSM powercycle test using a pre-integrated WT build (master):

  • WT git hash: a2d6c93989e96fccc889a7acb1a0cd36927e686e
  • Mongo git hash: a06c27a9887c9dd8fc43d9e7d8727a1840292911

I observed this failure:

2015-07-30T10:38:07.865-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
2015-07-30T10:38:07.923-0400 W -        [initandlisten] Detected unclean shutdown - /home/jonathan/pt-317-wt-lsm-100k/data/wiredTiger/mongod.lock is not empty.
2015-07-30T10:38:07.923-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2015-07-30T10:38:07.923-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-07-30T10:38:10.351-0400 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2015-07-30T10:38:10.384-0400 E STORAGE  [initandlisten] WiredTiger (0) [1438267090:384523][2598:0x7feb55da7cc0], file:collection-7-6115317438928344703-000656.bf, WT_CURSOR.prev: read checksum error for 8192B block at offset 36864: calculated block checksum of 4294396099 doesn't match expected checksum of 1369079895
2015-07-30T10:38:10.384-0400 E STORAGE  [initandlisten] WiredTiger (0) [1438267090:384560][2598:0x7feb55da7cc0], file:collection-7-6115317438928344703-000656.bf, WT_CURSOR.prev: collection-7-6115317438928344703-000656.bf: encountered an illegal file format or internal value
2015-07-30T10:38:10.384-0400 E STORAGE  [initandlisten] WiredTiger (-31804) [1438267090:384587][2598:0x7feb55da7cc0], file:collection-7-6115317438928344703-000656.bf, WT_CURSOR.prev: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-07-30T10:38:10.384-0400 I -        [initandlisten] Fatal Assertion 28558
2015-07-30T10:38:10.419-0400 I CONTROL  [initandlisten]
 0x1185112 0x1133b34 0x111f465 0xf48d43 0x184ba0c 0x184bbad 0x184bf94 0x179bcb4 0x17b639a 0x17bb178 0x17b821b 0x17ccae2 0x17cd921 0x17b5067 0x17e0580 0x184a966 0x184acc4 0x184ae1b 0x17ed918 0x1848e2a 0x179e050 0x179e59c 0x1809b28 0x180d627 0xf3cc3f 0xf38a2d 0xf3248c 0xe95f34 0xe9b038 0xf30fdc 0xe5a8b9 0x8b8fb0 0x8bdc64 0x7feb549b8af5 0x8b7209
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"D85112"},{"b":"400000","o":"D33B34"},{"b":"400000","o":"D1F465"},{"b":"400000","o":"B48D43"},{"b":"400000","o":"144BA0C"},{"b":"400000","o":"144BBAD"},{"b":"400000","o":"144BF94"},{"b":"400000","o":"139BCB4"},{"b":"400000","o":"13B639A"},{"b":"400000","o":"13BB178"},{"b":"400000","o":"13B821B"},{"b":"400000","o":"13CCAE2"},{"b":"400000","o":"13CD921"},{"b":"400000","o":"13B5067"},{"b":"400000","o":"13E0580"},{"b":"400000","o":"144A966"},{"b":"400000","o":"144ACC4"},{"b":"400000","o":"144AE1B"},{"b":"400000","o":"13ED918"},{"b":"400000","o":"1448E2A"},{"b":"400000","o":"139E050"},{"b":"400000","o":"139E59C"},{"b":"400000","o":"1409B28"},{"b":"400000","o":"140D627"},{"b":"400000","o":"B3CC3F"},{"b":"400000","o":"B38A2D"},{"b":"400000","o":"B3248C"},{"b":"400000","o":"A95F34"},{"b":"400000","o":"A9B038"},{"b":"400000","o":"B30FDC"},{"b":"400000","o":"A5A8B9"},{"b":"400000","o":"4B8FB0"},{"b":"400000","o":"4BDC64"},{"b":"7FEB54997000","o":"21AF5"},{"b":"400000","o":"4B7209"}],"processInfo":{ "mongodbVersion" : "3.1.7-pre-", "gitVersion" : "a06c27a9887c9dd8fc43d9e7d8727a1840292911", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-229.4.2.el7.x86_64", "version" : "#1 SMP Wed May 13 10:06:09 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF7F0FE000", "elfType" : 3 }, { "b" : "7FEB55997000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FEB55793000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FEB5548C000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FEB5518A000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FEB54F74000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FEB54D58000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FEB54997000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FEB55B9F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1185112]
 mongod(_ZN5mongo10logContextEPKc+0x134) [0x1133b34]
 mongod(_ZN5mongo13fassertFailedEi+0x55) [0x111f465]
 mongod(+0xB48D43) [0xf48d43]
 mongod(__wt_eventv+0x40C) [0x184ba0c]
 mongod(__wt_err+0x8D) [0x184bbad]
 mongod(__wt_panic+0x24) [0x184bf94]
 mongod(__wt_bm_read+0x74) [0x179bcb4]
 mongod(__wt_bt_read+0x1DA) [0x17b639a]
 mongod(__wt_cache_read+0x98) [0x17bb178]
 mongod(__wt_page_in_func+0x50B) [0x17b821b]
 mongod(+0x13CCAE2) [0x17ccae2]
 mongod(__wt_tree_walk+0x311) [0x17cd921]
 mongod(__wt_btree_open+0x1447) [0x17b5067]
 mongod(__wt_conn_btree_open+0x140) [0x17e0580]
 mongod(__wt_session_get_btree+0x166) [0x184a966]
 mongod(__wt_session_get_btree+0x4C4) [0x184acc4]
 mongod(__wt_session_get_btree_ckpt+0xBB) [0x184ae1b]
 mongod(__wt_curfile_open+0x248) [0x17ed918]
 mongod(__wt_open_cursor+0x27A) [0x1848e2a]
 mongod(+0x139E050) [0x179e050]
 mongod(__wt_bloom_open+0x4C) [0x179e59c]
 mongod(+0x1409B28) [0x1809b28]
 mongod(+0x140D627) [0x180d627]
 mongod(_ZN5mongo21WiredTigerRecordStore6Cursor4nextEv+0x5F) [0xf3cc3f]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_bllPNS_28CappedDocumentDeleteCallbackEPNS_20WiredTigerSizeStorerE+0x31D) [0xf38a2d]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x14C) [0xf3248c]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb+0x204) [0xe95f34]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x648) [0xe9b038]
 mongod(+0xB30FDC) [0xf30fdc]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x589) [0xe5a8b9]
 mongod(_ZN5mongo13initAndListenEi+0x160) [0x8b8fb0]
 mongod(main+0x154) [0x8bdc64]
 libc.so.6(__libc_start_main+0xF5) [0x7feb549b8af5]
 mongod(+0x4B7209) [0x8b7209]
-----  END BACKTRACE  -----
2015-07-30T10:38:10.419-0400 I -        [initandlisten]
 
***aborting after fassert() failure

Comment by Alexander Gorrod [ 23/Jul/15 ]

I've been running the powertest script against LSM with the latest fixes for SERVER-19274 and no longer see the error reported in this issue.

Comment by Alexander Gorrod [ 28/May/15 ]

rui.zhangIs this one reproducible? There have been a number of fixes for LSM and recovery since the issue was raised.

Comment by David Hows [ 01/Apr/15 ]

Just tried to reproduce this.

There was a large delay in terminating the MongoD after inserting the 160M documents, but trying to start a new instance while the existing one was still terminating gave me the expected errors about open lock file.

Subsequent reboots were quick and no errors.

rui.zhang, can you confirm if there are other steps I should be taking? Or if this is resolved?

Comment by Rui Zhang (Inactive) [ 27/Oct/14 ]

I re-test, it seems the "Fatal assertion" error is gone, I am getting "lsm-worker: Error in LSM worker thread 2: No such file or directory" error all the time. Will change title

Also found out that there are quite some files missed/removed after a few restarts

This is the DB which can start correctly with WT

[slave-7] ➜  dbs-wt  ls | wc
    716     716   21011
[slave-7] ➜  dbs-wt  du -h .
143G    .
[slave-7] ➜  dbs-wt  ls -la | awk '{total = total + $5}END{print total}'
152951496430

sometime during test, the directory just became much smaller, there are total (716-127) =589 less files, and size reduced from 143G to 83G. With the following on disk DB files, I am getting "lsm-worker: Error in LSM worker thread 2: No such file or directory"

[slave-7] ➜  dbs-wt  ls | wc
    127     127    6285
[slave-7] ➜  dbs-wt  du -h .
83G     .
[slave-7] ➜  dbs-wt  ls -la | awk '{total = total + $5}END{print total}'
88840155633

comparing files names, it seems most files missing are WiredTigerLog.0000xxx files, but there are one .lsm file missing (or changed name)

➜  db-corruption  diff good.1.sorted bad.1.sorted | grep -v WiredTigerLog
6,595c6
---
608c19
< collection--7091264320041286312-14-000323.lsm
---
> collection--7091264320041286312-14-000324.bf

here is the scripted I used to start wiredtiger

#!/bin/bash
 
set -x
 
killall mongod
sleep 5
 
DATADISK=/data/ssd-sdc
DBDIR=$DATADISK/dbs-wt
LOGDIR=$DATADISK/logs-wt/mongod.log
 
ulimit -f unlimited
ulimit -t unlimited
ulimit -v unlimited
ulimit -m unlimited
ulimit -n 64000
ulimit -u 64000
ulimit -a
 
date
LD_LIBRARY_PATH=$WT_INSTALL/lib numactl --physcpubind=0-3 --membind=0 -- ./bin/mongod --dbpath=$DBDIR --storageEngine=wiredtiger --logpath=$LOGDIR --fork
date

Comment by Rui Zhang (Inactive) [ 21/Oct/14 ]

this are the SHA I tested, shall already have the fix:

mongod: d0f6dd1ebc338dc96a672057289c5fef56ecad0b
wt: 525b9969876069fa4669cebe11f2ee89368f907b
(add in the Jira environment, seems not very visible).

I will get today's latest and give it another try.

Comment by Daniel Pasette (Inactive) [ 21/Oct/14 ]

Thanks. This should be fixed by this commit.
https://github.com/wiredtiger/wiredtiger/commit/ed8b6bd69e8609215cdc36b8869a5464c0519313

Please verify and close.

Comment by Rui Zhang (Inactive) [ 21/Oct/14 ]

The issue is related to DB files on disk, change dbpath to an empty folder, mongod+wt starts fine.

most time I got following Fatal assertion 28519 InternalError 16

 
 
[slave-7] ➜  wt  more /data/ssd-sdc/logs/mongod.log
2014-10-21T19:53:33.580+0000 I -        [initandlisten] MongoDB starting : pid=1232 port=27017 dbpath=/data/ssd-sdc/dbs 64-bit host=slave-7.perf.ny.cbi.10gen.cc
2014-10-21T19:53:33.580+0000 I -        [initandlisten]
2014-10-21T19:53:33.580+0000 I -        [initandlisten] ** NOTE: This is a development version (2.7.8-pre-) of MongoDB.
2014-10-21T19:53:33.580+0000 I -        [initandlisten] **       Not recommended for production.
2014-10-21T19:53:33.580+0000 I -        [initandlisten]
2014-10-21T19:53:33.580+0000 I -        [initandlisten]
2014-10-21T19:53:33.580+0000 I -        [initandlisten] ** WARNING: You are running on a NUMA machine.
2014-10-21T19:53:33.580+0000 I -        [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2014-10-21T19:53:33.580+0000 I -        [initandlisten] **              numactl --interleave=all mongod [other options]
2014-10-21T19:53:33.580+0000 I -        [initandlisten]
2014-10-21T19:53:33.580+0000 I -        [initandlisten] db version v2.7.8-pre-
2014-10-21T19:53:33.580+0000 I -        [initandlisten] git version: d0f6dd1ebc338dc96a672057289c5fef56ecad0b
2014-10-21T19:53:33.580+0000 I -        [initandlisten] build info: Linux slave-7.perf.ny.cbi.10gen.cc 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
2014-10-21T19:53:33.580+0000 I -        [initandlisten] allocator: tcmalloc
2014-10-21T19:53:33.580+0000 I -        [initandlisten] options: { processManagement: { fork: true }, storage: { dbPath: "/data/ssd-sdc/dbs", engine: "wiredtiger" }, systemLog: { destination: "file", path: "/
data/ssd-sdc/logs/mongod.log" } }
2014-10-21T19:53:38.582+0000 E STORAGE  [initandlisten] WiredTiger (16) [1413921218:581971][123240a7e9b4a17f0000], wiredtiger_open: WiredTiger database is already being managed by another process: Device or r
esource busy
2014-10-21T19:53:38.582+0000 I -        [initandlisten] Fatal assertion 28519 InternalError 16: Device or resource busy
2014-10-21T19:53:38.586+0000 I -        [initandlisten]
 0xfab9fe 0xf5d76f 0xf47769 0xddbef9 0xddab99 0xaa3a4d 0x7f0d5a 0x7f1c49 0x7f2849 0x7fa1b3866d5d 0x7edcb9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BAB9FE"},{"b":"400000","o":"B5D76F"},{"b":"400000","o":"B47769"},{"b":"400000","o":"9DBEF9"},{"b":"400000","o":"9DAB99"},{"b":"400000","o":"6A3A4D"},{"b":"400000","o":"3F0D5A"
},{"b":"400000","o":"3F1C49"},{"b":"400000","o":"3F2849"},{"b":"7FA1B3848000","o":"1ED5D"},{"b":"400000","o":"3EDCB9"}],"processInfo":{ "mongodbVersion" : "2.7.8-pre-", "gitVersion" : "d0f6dd1ebc338dc96a67205
7289c5fef56ecad0b", "uname" : { "sysname" : "Linux", "release" : "2.6.32-431.el6.x86_64", "version" : "#1 SMP Fri Nov 22 03:15:09 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000"
, "buildId" : "684FF49BCA63CF12CC4B5E482F0DF1A62471CD40" }, { "b" : "7FFFFFEFF000", "elfType" : 3, "buildId" : "81A81BE2E44C93640ADEDB62ADC93A47F4A09DD1" }, { "b" : "7FA1B49A5000", "path" : "/home/rzhang/wt/L
OCAL_INSTALL/lib/libwiredtiger-2.4.1.so", "elfType" : 3, "buildId" : "65CA56255224D2EDCCAC87C9A73F30F6318C1065" }, { "b" : "7FA1B4788000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "211321
F78CA244BE2B2B1B8584B460F9933BA76B" }, { "b" : "7FA1B4580000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "B26528BF6C0636AC1CAE5AC50BDBC07E60851DF4" }, { "b" : "7FA1B437C000", "path" : "/lib64/l
ibdl.so.2", "elfType" : 3, "buildId" : "AFC7448F2F2F6ED4E5BC82B1BD8A7320B84A9D48" }, { "b" : "7FA1B4076000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "ED99110E629209C5CA6C0ED704F2C5CE3
171513A" }, { "b" : "7FA1B3DF2000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "98B028A725D6E93253F25DF00B794DFAA66A3145" }, { "b" : "7FA1B3BDC000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3
, "buildId" : "A44499D29B114A5366CD72DD4883958495AC1C1D" }, { "b" : "7FA1B3848000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "59640F8CD5A70CF0391A7C64275D84336935E6AA" }, { "b" : "7FA1B4C8B000"
, "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "57BF668F99B7F5917B8D55FBB645173C9A644575" }, { "b" : "7FA1B3632000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC0
4A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7FA1B342C000", "path" : "/home/rzhang/wt/LOCAL_INSTALL/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "04D48EBBAB40DD47A25A411DB1F2CD2F55EE94AE" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x2E) [0xfab9fe]
 mongod(_ZN5mongo10logContextEPKc+0xDF) [0xf5d76f]
 mongod(_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE+0xD9) [0xf47769]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_+0x549) [0xddbef9]
 mongod(+0x9DAB99) [0xddab99]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x17D) [0xaa3a4d]
 mongod(_ZN5mongo13initAndListenEi+0x52A) [0x7f0d5a]
 mongod(+0x3F1C49) [0x7f1c49]
 mongod(main+0x9) [0x7f2849]
 libc.so.6(__libc_start_main+0xFD) [0x7fa1b3866d5d]
 mongod(+0x3EDCB9) [0x7edcb9]
-----  END BACKTRACE  -----
2014-10-21T19:53:38.586+0000 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2014-10-21T19:53:38.590+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0xfab9fe 0xfab488 0xfab771 0x7fa1b387a6b0 0x7fa1b387a635 0x7fa1b387be15 0xf477f7 0xddbef9 0xddab99 0xaa3a4d 0x7f0d5a 0x7f1c49 0x7f2849 0x7fa1b3866d5d 0x7edcb9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BAB9FE"},{"b":"400000","o":"BAB488"},{"b":"400000","o":"BAB771"},{"b":"7FA1B3848000","o":"326B0"},{"b":"7FA1B3848000","o":"32635"},{"b":"7FA1B3848000","o":"33E15"},{"b":"40000
0","o":"B477F7"},{"b":"400000","o":"9DBEF9"},{"b":"400000","o":"9DAB99"},{"b":"400000","o":"6A3A4D"},{"b":"400000","o":"3F0D5A"},{"b":"400000","o":"3F1C49"},{"b":"400000","o":"3F2849"},{"b":"7FA1B3848000","o"
:"1ED5D"},{"b":"400000","o":"3EDCB9"}],"processInfo":{ "mongodbVersion" : "2.7.8-pre-", "gitVersion" : "d0f6dd1ebc338dc96a672057289c5fef56ecad0b", "uname" : { "sysname" : "Linux", "release" : "2.6.32-431.el6.
x86_64", "version" : "#1 SMP Fri Nov 22 03:15:09 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "684FF49BCA63CF12CC4B5E482F0DF1A62471CD40" }, { "b" : "7FFFFFEFF000
", "elfType" : 3, "buildId" : "81A81BE2E44C93640ADEDB62ADC93A47F4A09DD1" }, { "b" : "7FA1B49A5000", "path" : "/home/rzhang/wt/LOCAL_INSTALL/lib/libwiredtiger-2.4.1.so", "elfType" : 3, "buildId" : "65CA5625522
4D2EDCCAC87C9A73F30F6318C1065" }, { "b" : "7FA1B4788000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "211321F78CA244BE2B2B1B8584B460F9933BA76B" }, { "b" : "7FA1B4580000", "path" : "/lib64/l
ibrt.so.1", "elfType" : 3, "buildId" : "B26528BF6C0636AC1CAE5AC50BDBC07E60851DF4" }, { "b" : "7FA1B437C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "AFC7448F2F2F6ED4E5BC82B1BD8A7320B84A9D48"
 }, { "b" : "7FA1B4076000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "ED99110E629209C5CA6C0ED704F2C5CE3171513A" }, { "b" : "7FA1B3DF2000", "path" : "/lib64/libm.so.6", "elfType" : 3, "
buildId" : "98B028A725D6E93253F25DF00B794DFAA66A3145" }, { "b" : "7FA1B3BDC000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A44499D29B114A5366CD72DD4883958495AC1C1D" }, { "b" : "7FA1B3848000
", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "59640F8CD5A70CF0391A7C64275D84336935E6AA" }, { "b" : "7FA1B4C8B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "57BF668F99B
7F5917B8D55FBB645173C9A644575" }, { "b" : "7FA1B3632000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7FA1B342C000", "path" : "/home/rzhang/w
t/LOCAL_INSTALL/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "04D48EBBAB40DD47A25A411DB1F2CD2F55EE94AE" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x2E) [0xfab9fe]
 mongod(+0xBAB488) [0xfab488]
 mongod(+0xBAB771) [0xfab771]
 libc.so.6(+0x326B0) [0x7fa1b387a6b0]
 libc.so.6(gsignal+0x35) [0x7fa1b387a635]
 libc.so.6(abort+0x175) [0x7fa1b387be15]
 mongod(_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE+0x167) [0xf477f7]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_+0x549) [0xddbef9]
 mongod(+0x9DAB99) [0xddab99]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x17D) [0xaa3a4d]
 mongod(_ZN5mongo13initAndListenEi+0x52A) [0x7f0d5a]
 mongod(+0x3F1C49) [0x7f1c49]
 mongod(main+0x9) [0x7f2849]
 libc.so.6(__libc_start_main+0xFD) [0x7fa1b3866d5d]
 mongod(+0x3EDCB9) [0x7edcb9]
-----  END BACKTRACE  -----
[slave-7] ➜  wt

also saw following error (rate compare to Fatal assertion error)

2014-10-21T19:33:27.552+0000 E STORAGE  WiredTiger (2) [1413920007:552697][198720027f109de7f0000], lsm-worker: Error in LSM worker thread 2: No such file or directory

complete log:

+ date
Tue Oct 21 19:33:12 UTC 2014
+ LD_LIBRARY_PATH=/home/rzhang/wt/LOCAL_INSTALL/lib
+ numactl --physcpubind=0-3 --membind=0 -- ./bin/mongod --dbpath=/data/ssd-sdc/dbs --storageEngine=wiredtiger --logpath=/data/ssd-sdc/logs/mongod.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 19872
child process started successfully, parent exiting
+ date
Tue Oct 21 19:36:30 UTC 2014
[slave-7] ➜  wt
[slave-7] ➜  wt
[slave-7] ➜  wt
[slave-7] ➜  wt  more /data/ssd-sdc/logs/mongod.log
2014-10-21T19:33:12.943+0000 I -        [initandlisten] MongoDB starting : pid=19872 port=27017 dbpath=/data/ssd-sdc/dbs 64-bit host=slave-7.perf.ny.cbi.10gen.cc
2014-10-21T19:33:12.943+0000 I -        [initandlisten]
2014-10-21T19:33:12.943+0000 I -        [initandlisten] ** NOTE: This is a development version (2.7.8-pre-) of MongoDB.
2014-10-21T19:33:12.943+0000 I -        [initandlisten] **       Not recommended for production.
2014-10-21T19:33:12.943+0000 I -        [initandlisten]
2014-10-21T19:33:12.943+0000 I -        [initandlisten]
2014-10-21T19:33:12.943+0000 I -        [initandlisten] ** WARNING: You are running on a NUMA machine.
2014-10-21T19:33:12.943+0000 I -        [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2014-10-21T19:33:12.943+0000 I -        [initandlisten] **              numactl --interleave=all mongod [other options]
2014-10-21T19:33:12.943+0000 I -        [initandlisten]
2014-10-21T19:33:12.943+0000 I -        [initandlisten] db version v2.7.8-pre-
2014-10-21T19:33:12.943+0000 I -        [initandlisten] git version: d0f6dd1ebc338dc96a672057289c5fef56ecad0b
2014-10-21T19:33:12.943+0000 I -        [initandlisten] build info: Linux slave-7.perf.ny.cbi.10gen.cc 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
2014-10-21T19:33:12.943+0000 I -        [initandlisten] allocator: tcmalloc
2014-10-21T19:33:12.943+0000 I -        [initandlisten] options: { processManagement: { fork: true }, storage: { dbPath: "/data/ssd-sdc/dbs", engine: "wiredtiger" }, systemLog: { destination: "file", path: "/
data/ssd-sdc/logs/mongod.log" } }
2014-10-21T19:33:27.533+0000 E STORAGE  WiredTiger (2) [1413920007:533708][198720037910ade7f0000], lsm-worker: Error in LSM worker thread 1: No such file or directory
2014-10-21T19:33:27.542+0000 E STORAGE  WiredTiger (2) [1413920007:542634][1987200175109de7f0000], lsm-worker: Error in LSM worker thread 3: No such file or directory
2014-10-21T19:33:27.552+0000 E STORAGE  WiredTiger (2) [1413920007:552697][198720027f109de7f0000], lsm-worker: Error in LSM worker thread 2: No such file or directory
[slave-7] ➜  wt
 

Generated at Thu Feb 08 03:39:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.