[SERVER-2655] !haveJournalFiles() assertion in repl3 test Created: 01/Mar/11  Updated: 12/Jul/16  Resolved: 09/Mar/11

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 1.8.0-rc2

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

<http://buildbot.mongodb.org/builders/Linux%2064-bit%20DUR%20ON/builds/267/steps/test_4/logs/stdio>

m31001| repl: time diff: 2sec
m31001| repl: tailing: 0
m31001| repl: data too stale, halting replication
m31001| Sun Feb 27 12:57:19 [replslave] caught SyncException
m31001| Sun Feb 27 12:57:19 [replslave] repl: sleep 10sec before next pass
m31000| Sun Feb 27 12:57:29 [initandlisten] connection accepted from 127.0.0.1:55772 #5
m31001| Sun Feb 27 12:57:29 [replslave] resync: dropping database jstests_repl3test
m31001| Sun Feb 27 12:57:29 [replslave] removeJournalFiles
m31001| Sun Feb 27 12:57:29 [dur] lsn set 11151
m31001| Sun Feb 27 12:57:29 [replslave] jstests_repl3test Assertion failure !haveJournalFiles() db/dur.cpp 648
m31001| 0x5255c1 0x535d01 0x6eee31 0x639296 0x649b0b 0x649e98 0x649f64 0x64a0ac 0x64a605 0x2ac7a91ed230 0x2ac7a8ac1971 0x2ac7aa55f92d
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo12sayDbContextEPKc+0xb1) [0x5255c1]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo8assertedEPKcS1_j+0xc1) [0x535d01]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo12dropDatabaseESs+0xc1) [0x6eee31]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(ZN5mongo10ReplSource10resyncDropEPKcS2+0xc6) [0x639296]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo10ReplSource11forceResyncEPKc+0x47b) [0x649b0b]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo10ReplSource15forceResyncDeadEPKc+0x68) [0x649e98]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo10ReplSource24throttledForceResyncDeadEPKc+0x34) [0x649f64]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo8replMainEv+0x11c) [0x64a0ac]
m31001| /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/mongod(_ZN5mongo15replSlaveThreadEv+0x175) [0x64a605]
m31001| /usr/lib/libboost_thread.so.1.42.0(thread_proxy+0x60) [0x2ac7a91ed230]
m31001| /lib/libpthread.so.0(+0x7971) [0x2ac7a8ac1971]
m31001| /lib/libc.so.6(clone+0x6d) [0x2ac7aa55f92d]
m31001| Sun Feb 27 12:57:29 [replslave] Assertion in replSlaveThread(): sleeping 5 minutes before retry
m31000| Sun Feb 27 12:57:29 [conn4] end connection 127.0.0.1:55771
m31000| Sun Feb 27 12:57:29 [conn5] end connection 127.0.0.1:55772
Sun Feb 27 12:57:49 [dur] lsn set 656219
assert.soon failed: function () {
return s.getDB(baseName).a.find().count() == count;
}, msg:undefined
Error("Printing Stack Trace")@:0
()@shell/utils.js:21
("assert.soon failed: function ()

{\n return s.getDB(baseName).a.find().count() == count;\n}

, msg:undefined")@shell/utils.js:32
((function ()

{return s.getDB(baseName).a.find().count() == count;}

))@shell/utils.js:118
(1001)@/home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/jstests/repl/repl3.js:6
(15)@/home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/jstests/repl/repl3.js:36
@/home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/jstests/repl/repl3.js:46

Sun Feb 27 12:58:03 uncaught exception: assert.soon failed: function () {
return s.getDB(baseName).a.find().count() == count;
}, msg:undefined
failed to load: /home/yellow/buildslave/Linux_64bit_DUR_ON/mongo/jstests/repl/repl3.js



 Comments   
Comment by auto [ 10/Mar/11 ]

Author:

{u'login': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: Don't rotate and open a journal file if one isn't already opened SERVER-2655 180
https://github.com/mongodb/mongo/commit/e92bdfba29dd9085689eecb5d38e0fa0bc11957f

Comment by auto [ 09/Mar/11 ]

Author:

{u'login': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: Don't rotate and open a journal file if one isn't already opened SERVER-2655 180
https://github.com/mongodb/mongo/commit/b2927d98603a48f4437dd39116e78bf023b4ee8c

Comment by auto [ 09/Mar/11 ]

Author:

{u'login': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: Don't rotate and open a journal file if one isn't already opened SERVER-2655 180
https://github.com/mongodb/mongo/commit/e92bdfba29dd9085689eecb5d38e0fa0bc11957f

Comment by Tony Hannan [ 08/Mar/11 ]

I see this error at the end of a --repair of a large database (10x RAM) on a single server with --dur

mongod log:
...
Tue Mar 8 20:59:57 [initandlisten] getmore test.foo cid:483941047588692145 getMore: {} bytes:4718984 nreturned:6 840ms
Tue Mar 8 20:59:58 [initandlisten] getmore test.foo cid:483941047588692145 getMore: {} bytes:4718984 nreturned:6 156ms
Tue Mar 8 20:59:59 [initandlisten] getmore test.foo cid:483941047588692145 getMore: {} bytes:3145996 nreturned:4 149ms
Tue Mar 8 20:59:59 [initandlisten] building new index on

{ _id: 1 }

for test.foo
2200/6000 36%
5400/6000 90%
Tue Mar 8 21:00:21 [initandlisten] external sort used : 0 files in 22 secs
Tue Mar 8 21:00:22 [initandlisten] done building bottom layer, going to commit
Tue Mar 8 21:00:22 [initandlisten] done for 6000 records 22.817secs
Tue Mar 8 21:00:22 [dur] lsn set 0
Tue Mar 8 21:00:22 [dur] lsn set 0
Tue Mar 8 21:00:23 [initandlisten] removeJournalFiles
Tue Mar 8 21:00:23 [dur] lsn set 0
Tue Mar 8 21:00:23 [initandlisten] Assertion failure !haveJournalFiles() db/dur.cpp 630
0x55133e 0x562d41 0x781e8e 0x8a6a78 0x8a79b4 0x8aa65c 0x8ab2c8 0x8b0edf 0x7f7450b9d1a6 0x4e1059
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x55133e]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x111) [0x562d41]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x6fe) [0x781e8e]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x8a6a78]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod [0x8a79b4]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo14_initAndListenEiPKc+0x39c) [0x8aa65c]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo13initAndListenEiPKc+0x18) [0x8ab2c8]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(main+0x5aaf) [0x8b0edf]
/lib/libc.so.6(__libc_start_main+0xe6) [0x7f7450b9d1a6]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(__gxx_personality_v0+0x3b1) [0x4e1059]
Tue Mar 8 21:00:24 [initandlisten] exception in initAndListen std::exception: assertion db/dur.cpp:630, terminating
Tue Mar 8 21:00:24 dbexit:
Tue Mar 8 21:00:24 [initandlisten] shutdown: going to close listening sockets...
Tue Mar 8 21:00:24 [initandlisten] shutdown: going to flush diaglog...
Tue Mar 8 21:00:24 [initandlisten] shutdown: going to close sockets...
Tue Mar 8 21:00:24 [initandlisten] shutdown: waiting for fs preallocator...
Tue Mar 8 21:00:24 [initandlisten] shutdown: lock for final commit...
Tue Mar 8 21:00:24 [initandlisten] shutdown: final commit...
Tue Mar 8 21:00:24 [initandlisten] shutdown: closing all files...
Tue Mar 8 21:00:24 closeAllFiles() finished
Tue Mar 8 21:00:24 [initandlisten] shutdown: journalCleanup...
Tue Mar 8 21:00:24 [initandlisten] removeJournalFiles
Tue Mar 8 21:00:24 [initandlisten] shutdown: removing fs lock...
Tue Mar 8 21:00:24 dbexit: really exiting now
mongod: /opt/extra/include/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:25: boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*): Assertion `!pthread_mutex_lock(m)' failed.
Tue Mar 8 21:00:24 Got signal: 6 (Aborted).

Tue Mar 8 21:00:24 Backtrace:
0x8a54b9 0x7f7450bb0f60 0x7f7450bb0ed5 0x7f7450bb23f3 0x7f7450ba9dc9 0x572c59 0x62f57e 0x630023 0x8b8d80 0x7f745167efc7 0x7f7450c4e5ad
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a54b9]
/lib/libc.so.6 [0x7f7450bb0f60]
/lib/libc.so.6(gsignal+0x35) [0x7f7450bb0ed5]
/lib/libc.so.6(abort+0x183) [0x7f7450bb23f3]
/lib/libc.so.6(__assert_fail+0xe9) [0x7f7450ba9dc9]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo9NotifyAll9notifyAllEv+0xb9) [0x572c59]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod [0x62f57e]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo3dur9durThreadEv+0x223) [0x630023]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(thread_proxy+0x80) [0x8b8d80]
/lib/libpthread.so.0 [0x7f745167efc7]
/lib/libc.so.6(clone+0x6d) [0x7f7450c4e5ad]

Generated at Thu Feb 08 03:00:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.