[SERVER-5465] jstests/sharding/remove1.js failed on Linux 64 dur off Created: 30/Mar/12  Updated: 15/May/12  Resolved: 15/May/12

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.1.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andy Schwerin Assignee: Eliot Horowitz (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-5446 Intermittent failure to remove local ... Closed
Operating System: ALL
Participants:

 Description   

May be sporadic. Need to understand the meaning of the failure mode.

http://buildbot.mongodb.org/builders/Linux%2064-bit%20debug%20dur%20off/builds/1024/steps/test_9/logs/stdio

Relevant section:

        Test : remove1.js ...
      Command : /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongo --port 27999 --nodb /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/jstests/sharding/remove1.js --eval TestData = new Object();TestData.testPath = "/home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/jstests/sharding/remove1.js";TestData.testFile = "remove1.js";TestData.testName = "remove1";TestData.noJournal = false;TestData.noJournalPrealloc = false;TestData.auth = false;TestData.keyFile = null;TestData.keyFileData = null;
MongoDB shell version: 2.1.1-pre-
null
Resetting db path '/data/db/remove_shard10'
shell: started program /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod --port 30000 --dbpath /data/db/remove_shard10
 m30000| db level locking enabled: 0
 m30000| Fri Mar 30 12:13:43 [initandlisten] MongoDB starting : pid=11745 port=30000 dbpath=/data/db/remove_shard10 64-bit host=tp1.10gen.cc
 m30000| Fri Mar 30 12:13:43 [initandlisten] _DEBUG build (which is slower)
 m30000| Fri Mar 30 12:13:43 [initandlisten] 
 m30000| Fri Mar 30 12:13:43 [initandlisten] ** NOTE: This is a development version (2.1.1-pre-) of MongoDB.
 m30000| Fri Mar 30 12:13:43 [initandlisten] **       Not recommended for production.
 m30000| Fri Mar 30 12:13:43 [initandlisten] 
 m30000| Fri Mar 30 12:13:43 [initandlisten] db version v2.1.1-pre-, pdfile version 4.5
 m30000| Fri Mar 30 12:13:43 [initandlisten] git version: 5455f1d95c232c92647e3e0cfda4f52b28a08ef7
 m30000| Fri Mar 30 12:13:43 [initandlisten] build info: Linux tp1.10gen.cc 2.6.31-14-server #48-Ubuntu SMP Fri Oct 16 15:07:34 UTC 2009 x86_64 BOOST_LIB_VERSION=1_49
 m30000| Fri Mar 30 12:13:43 [initandlisten] options: { dbpath: "/data/db/remove_shard10", port: 30000 }
 m30000| Fri Mar 30 12:13:43 [initandlisten] opening db:  local
 m30000| Fri Mar 30 12:13:43 [websvr] admin web console waiting for connections on port 31000
 m30000| Fri Mar 30 12:13:43 [initandlisten] waiting for connections on port 30000
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53019 #1 (1 connection now open)
Resetting db path '/data/db/remove_shard11'
shell: started program /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod --port 30001 --dbpath /data/db/remove_shard11
 m30001| db level locking enabled: 0
 m30001| Fri Mar 30 12:13:43 [initandlisten] MongoDB starting : pid=11757 port=30001 dbpath=/data/db/remove_shard11 64-bit host=tp1.10gen.cc
 m30001| Fri Mar 30 12:13:43 [initandlisten] _DEBUG build (which is slower)
 m30001| Fri Mar 30 12:13:43 [initandlisten] 
 m30001| Fri Mar 30 12:13:43 [initandlisten] ** NOTE: This is a development version (2.1.1-pre-) of MongoDB.
 m30001| Fri Mar 30 12:13:43 [initandlisten] **       Not recommended for production.
 m30001| Fri Mar 30 12:13:43 [initandlisten] 
 m30001| Fri Mar 30 12:13:43 [initandlisten] db version v2.1.1-pre-, pdfile version 4.5
 m30001| Fri Mar 30 12:13:43 [initandlisten] git version: 5455f1d95c232c92647e3e0cfda4f52b28a08ef7
 m30001| Fri Mar 30 12:13:43 [initandlisten] build info: Linux tp1.10gen.cc 2.6.31-14-server #48-Ubuntu SMP Fri Oct 16 15:07:34 UTC 2009 x86_64 BOOST_LIB_VERSION=1_49
 m30001| Fri Mar 30 12:13:43 [initandlisten] options: { dbpath: "/data/db/remove_shard11", port: 30001 }
 m30001| Fri Mar 30 12:13:43 [initandlisten] opening db:  local
 m30001| Fri Mar 30 12:13:43 [websvr] admin web console waiting for connections on port 31001
 m30001| Fri Mar 30 12:13:43 [initandlisten] waiting for connections on port 30001
 m30001| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:47497 #1 (1 connection now open)
"localhost:30000"
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53022 #2 (2 connections now open)
ShardingTest remove_shard1 :
{
	"config" : "localhost:30000",
	"shards" : [
		connection to localhost:30000,
		connection to localhost:30001
	]
}
 m30000| Fri Mar 30 12:13:43 [conn2] opening db:  config
shell: started program /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos --port 30999 --configdb localhost:30000
 m30999| Fri Mar 30 12:13:43 warning: running with 1 config server should be done only for testing purposes and is not recommended for production
 m30999| Fri Mar 30 12:13:43 [mongosMain] MongoS version 2.1.1-pre- starting: pid=11771 port=30999 64-bit host=tp1.10gen.cc (--help for usage)
 m30999| Fri Mar 30 12:13:43 [mongosMain] _DEBUG build
 m30999| Fri Mar 30 12:13:43 [mongosMain] git version: 5455f1d95c232c92647e3e0cfda4f52b28a08ef7
 m30999| Fri Mar 30 12:13:43 [mongosMain] build info: Linux tp1.10gen.cc 2.6.31-14-server #48-Ubuntu SMP Fri Oct 16 15:07:34 UTC 2009 x86_64 BOOST_LIB_VERSION=1_49
 m30999| Fri Mar 30 12:13:43 [mongosMain] options: { configdb: "localhost:30000", port: 30999 }
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53023 #3 (3 connections now open)
 m30000| Fri Mar 30 12:13:43 [FileAllocator] allocating new datafile /data/db/remove_shard10/config.ns, filling with zeroes...
 m30000| Fri Mar 30 12:13:43 [FileAllocator] creating directory /data/db/remove_shard10/_tmp
 m30000| Fri Mar 30 12:13:43 [FileAllocator] done allocating datafile /data/db/remove_shard10/config.ns, size: 16MB,  took 0.016 secs
 m30000| Fri Mar 30 12:13:43 [FileAllocator] allocating new datafile /data/db/remove_shard10/config.0, filling with zeroes...
 m30000| Fri Mar 30 12:13:43 [FileAllocator] done allocating datafile /data/db/remove_shard10/config.0, size: 64MB,  took 0.018 secs
 m30000| Fri Mar 30 12:13:43 [conn2] datafileheader::init initializing /data/db/remove_shard10/config.0 n:0
 m30000| Fri Mar 30 12:13:43 [FileAllocator] allocating new datafile /data/db/remove_shard10/config.1, filling with zeroes...
 m30000| Fri Mar 30 12:13:43 [conn2] _DEBUG info _PAGEFAULTEXCEPTION is off
 m30000| Fri Mar 30 12:13:43 [conn2] build index config.settings { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn2] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53025 #4 (4 connections now open)
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53026 #5 (5 connections now open)
 m30000| Fri Mar 30 12:13:43 [conn5] build index config.version { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn5] build index done.  scanned 0 total records. 0 secs
 m30999| Fri Mar 30 12:13:43 [websvr] admin web console waiting for connections on port 31999
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.chunks { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [conn4] info: creating collection config.chunks on add index
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.chunks { ns: 1, min: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.chunks { ns: 1, shard: 1, min: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.chunks { ns: 1, lastmod: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.shards { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [conn4] info: creating collection config.shards on add index
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.shards { host: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30999| Fri Mar 30 12:13:43 [Balancer] about to contact config servers and shards
 m30999| Fri Mar 30 12:13:43 [mongosMain] waiting for connections on port 30999
 m30999| Fri Mar 30 12:13:43 [Balancer] config servers and shards contacted successfully
 m30999| Fri Mar 30 12:13:43 [Balancer] balancer id: tp1.10gen.cc:30999 started at Mar 30 12:13:43
 m30999| Fri Mar 30 12:13:43 [Balancer] created new distributed lock for balancer on localhost:30000 ( lock timeout : 900000, ping interval : 30000, process : 0 )
 m30000| Fri Mar 30 12:13:43 [conn5] build index config.mongos { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn5] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53027 #6 (6 connections now open)
 m30000| Fri Mar 30 12:13:43 [FileAllocator] done allocating datafile /data/db/remove_shard10/config.1, size: 128MB,  took 0.017 secs
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.lockpings { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.lockpings { ping: 1 }
 m30999| Fri Mar 30 12:13:43 [LockPinger] creating distributed lock ping thread for localhost:30000 and process tp1.10gen.cc:30999:1333127623:1804289383 (sleeping for 30000ms)
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 1 total records. 0 secs
 m30000| Fri Mar 30 12:13:43 [conn6] build index config.locks { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn6] build index done.  scanned 0 total records. 0 secs
 m30999| Fri Mar 30 12:13:43 [Balancer] distributed lock 'balancer/tp1.10gen.cc:30999:1333127623:1804289383' acquired, ts : 4f75e9c7cb03628d97f227c9
 m30999| Fri Mar 30 12:13:43 [Balancer] distributed lock 'balancer/tp1.10gen.cc:30999:1333127623:1804289383' unlocked. 
 m30999| Fri Mar 30 12:13:43 [mongosMain] connection accepted from 127.0.0.1:41026 #1 (1 connection now open)
ShardingTest undefined going to add shard : localhost:30000
 m30999| Fri Mar 30 12:13:43 [conn1] couldn't find database [admin] in config db
 m30000| Fri Mar 30 12:13:43 [conn4] build index config.databases { _id: 1 }
 m30000| Fri Mar 30 12:13:43 [conn4] build index done.  scanned 0 total records. 0 secs
 m30999| Fri Mar 30 12:13:43 [conn1] 	 put [admin] on: config:localhost:30000
 m30999| Fri Mar 30 12:13:43 [conn1] going to add shard: { _id: "shard0000", host: "localhost:30000" }
{ "shardAdded" : "shard0000", "ok" : 1 }
ShardingTest undefined going to add shard : localhost:30001
 m30001| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:47505 #2 (2 connections now open)
 m30999| Fri Mar 30 12:13:43 [conn1] going to add shard: { _id: "shard0001", host: "localhost:30001" }
{ "shardAdded" : "shard0001", "ok" : 1 }
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53030 #7 (7 connections now open)
 m30999| Fri Mar 30 12:13:43 [conn1] creating WriteBackListener for: localhost:30000 serverID: 4f75e9c7cb03628d97f227c8
 m30001| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:47507 #3 (3 connections now open)
 m30999| Fri Mar 30 12:13:43 [conn1] creating WriteBackListener for: localhost:30001 serverID: 4f75e9c7cb03628d97f227c8
 m30999| Fri Mar 30 12:13:43 [conn1] going to start draining shard: shard0000
 m30999| primaryLocalDoc: { _id: "local", primary: "shard0000" }
 m30000| Fri Mar 30 12:13:43 [initandlisten] connection accepted from 127.0.0.1:53032 #8 (8 connections now open)
 m30999| Fri Mar 30 12:13:43 [conn1] DROP DATABASE: needToMove
 m30999| Fri Mar 30 12:13:43 [conn1] DBConfig::dropDatabase: needToMove
 m30999| Fri Mar 30 12:13:43 [conn1] about to log metadata event: { _id: "tp1.10gen.cc-2012-03-30T17:13:43-0", server: "tp1.10gen.cc", clientAddr: "N/A", time: new Date(1333127623616), what: "dropDatabase.start", ns: "needToMove", details: {} }
 m30999| Fri Mar 30 12:13:43 [conn1]    DBConfig::dropDatabase: needToMove dropped sharded collections: 0
 m30000| Fri Mar 30 12:13:43 [conn4] opening db:  needToMove
 m30000| Fri Mar 30 12:13:43 [conn4] dropDatabase needToMove
 m30999| Fri Mar 30 12:13:43 [conn1] about to log metadata event: { _id: "tp1.10gen.cc-2012-03-30T17:13:43-1", server: "tp1.10gen.cc", clientAddr: "N/A", time: new Date(1333127623619), what: "dropDatabase", ns: "needToMove", details: {} }
 m30999| Fri Mar 30 12:13:43 [conn1] going to remove shard: shard0000
assert: supposed to be null (should have removed local db) was: { "_id" : "local", "partitioned" : false, "primary" : "shard0000" }
Error("Printing Stack Trace")@:0
()@src/mongo/shell/utils.js:37
("supposed to be null (should have removed local db) was: { \"_id\" : \"local\", \"partitioned\" : false, \"primary\" : \"shard0000\" }")@src/mongo/shell/utils.js:58
([object bson_object],"should have removed local db")@src/mongo/shell/utils.js:235
@/home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/jstests/sharding/remove1.js:18
 
Fri Mar 30 12:13:43 uncaught exception: supposed to be null (should have removed local db) was: { "_id" : "local", "partitioned" : false, "primary" : "shard0000" }
failed to load: /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/jstests/sharding/remove1.js
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] DBClientCursor::init call() failed
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] dev: lastError==0 won't report:DBClientBase::findN: transport error: localhost:30000 ns: admin.$cmd query: { writebacklisten: ObjectId('4f75e9c7cb03628d97f227c8') }
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] WriteBackListener exception : DBClientBase::findN: transport error: localhost:30000 ns: admin.$cmd query: { writebacklisten: ObjectId('4f75e9c7cb03628d97f227c8') }
 m30000| Fri Mar 30 12:13:43 got signal 15 (Terminated), will terminate after current cmd ends
 m30000| Fri Mar 30 12:13:43 [interruptThread] now exiting
 m30000| Fri Mar 30 12:13:43 dbexit: 
 m30000| Fri Mar 30 12:13:43 [interruptThread] shutdown: going to close listening sockets...
 m30000| Fri Mar 30 12:13:43 [interruptThread] closing listening socket: 35
 m30000| Fri Mar 30 12:13:43 [interruptThread] closing listening socket: 36
 m30000| Fri Mar 30 12:13:43 [interruptThread] closing listening socket: 37
 m30000| Fri Mar 30 12:13:43 [interruptThread] removing socket file: /tmp/mongodb-30000.sock
 m30000| Fri Mar 30 12:13:43 [interruptThread] shutdown: going to flush diaglog...
 m30000| Fri Mar 30 12:13:43 [interruptThread] shutdown: going to close sockets...
 m30000| Fri Mar 30 12:13:43 [conn6] end connection 127.0.0.1:53027 (7 connections now open)
 m30000| Fri Mar 30 12:13:43 [conn6] thread conn stack usage was 10176 bytes
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] Socket recv() errno:104 Connection reset by peer 127.0.0.1:30000
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] SocketException: remote: 127.0.0.1:30000 error: 9001 socket exception [1] server [127.0.0.1:30000] 
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] DBClientCursor::init call() failed
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] Assertion: 13632:couldn't get updated shard list from config server
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] dev: lastError==0 won't report:couldn't get updated shard list from config server
 m30999| 0x794b6f 0x9f83fb 0xb10700 0xb0f377 0x7b2009 0x87d0f9 0x8801b7 0x8800b7 0x880035 0x87ffc2 0x83b1f4 0x7f303193ea04 0x7f3030efcd4d 
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo15printStackTraceERSo+0x27) [0x794b6f]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo11msgassertedEiPKc+0xfd) [0x9f83fb]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo15StaticShardInfo6reloadEv+0x15c) [0xb10700]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo5Shard15reloadShardInfoEv+0x13) [0xb0f377]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo17WriteBackListener3runEv+0x1be9) [0x7b2009]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x35d) [0x87d0f9]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZNK5boost4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS3_9JobStatusEEEEclEPS3_S6_+0x7d) [0x8801b7]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5boost3_bi5list2INS0_5valueIPN5mongo13BackgroundJobEEENS2_INS_10shared_ptrINS4_9JobStatusEEEEEEclINS_4_mfi3mf1IvS4_S9_EENS0_5list0EEEvNS0_4typeIvEERT_RT0_i+0x79) [0x8800b7]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5boost3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS5_9JobStatusEEEEENS0_5list2INS0_5valueIPS5_EENSB_IS8_EEEEEclEv+0x3f) [0x880035]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x1e) [0x87ffc2]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos [0x83b1f4]
 m30999|  /lib/libpthread.so.0 [0x7f303193ea04]
 m30999|  /lib/libc.so.6(clone+0x6d) [0x7f3030efcd4d]
 m30999| Fri Mar 30 12:13:43 [WriteBackListener-localhost:30000] ERROR: backgroundjob WriteBackListener-localhost:30000error: couldn't get updated shard list from config server
 m30000| Fri Mar 30 12:13:43 [interruptThread] shutdown: waiting for fs preallocator...
 m30000| Fri Mar 30 12:13:43 [interruptThread] shutdown: closing all files...
 m30000| Fri Mar 30 12:13:43 [interruptThread] closeAllFiles() finished
 m30000| Fri Mar 30 12:13:43 [interruptThread] shutdown: removing fs lock...
 m30000| Fri Mar 30 12:13:43 dbexit: really exiting now
 m30001| Fri Mar 30 12:13:44 got signal 15 (Terminated), will terminate after current cmd ends
 m30001| Fri Mar 30 12:13:44 [interruptThread] now exiting
 m30001| Fri Mar 30 12:13:44 dbexit: 
 m30001| Fri Mar 30 12:13:44 [interruptThread] shutdown: going to close listening sockets...
 m30001| Fri Mar 30 12:13:44 [interruptThread] closing listening socket: 38
 m30001| Fri Mar 30 12:13:44 [interruptThread] closing listening socket: 39
 m30001| Fri Mar 30 12:13:44 [interruptThread] closing listening socket: 40
 m30001| Fri Mar 30 12:13:44 [interruptThread] removing socket file: /tmp/mongodb-30001.sock
 m30001| Fri Mar 30 12:13:44 [interruptThread] shutdown: going to flush diaglog...
 m30001| Fri Mar 30 12:13:44 [interruptThread] shutdown: going to close sockets...
 m30001| Fri Mar 30 12:13:44 [interruptThread] shutdown: waiting for fs preallocator...
 m30001| Fri Mar 30 12:13:44 [interruptThread] shutdown: closing all files...
 m30001| Fri Mar 30 12:13:44 [interruptThread] closeAllFiles() finished
 m30001| Fri Mar 30 12:13:44 [interruptThread] shutdown: removing fs lock...
 m30001| Fri Mar 30 12:13:44 dbexit: really exiting now
 m30001| Fri Mar 30 12:13:44 [interruptThread] thread interruptThread stack usage was 4560 bytes
 m30999| Fri Mar 30 12:13:44 [WriteBackListener-localhost:30001] DBClientCursor::init call() failed
 m30999| Fri Mar 30 12:13:44 [WriteBackListener-localhost:30001] dev: lastError==0 won't report:DBClientBase::findN: transport error: localhost:30001 ns: admin.$cmd query: { writebacklisten: ObjectId('4f75e9c7cb03628d97f227c8') }
 m30999| Fri Mar 30 12:13:44 [WriteBackListener-localhost:30001] WriteBackListener exception : DBClientBase::findN: transport error: localhost:30001 ns: admin.$cmd query: { writebacklisten: ObjectId('4f75e9c7cb03628d97f227c8') }
 m30999| Fri Mar 30 12:13:44 [WriteBackListener-localhost:30001] DBClientCursor::init call() failed
 m30999| Fri Mar 30 12:13:44 [WriteBackListener-localhost:30001] Assertion: 13632:couldn't get updated shard list from config server
 m30999| 0x794b6f 0x9f83fb 0xb10700 0xb0f377 0x7b2009 0x87d0f9 0x8801b7 0x8800b7 0x880035 0x87ffc2 0x83b1f4 0x7f303193ea04 0x7f3030efcd4d 
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo15printStackTraceERSo+0x27) [0x794b6f]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo11msgassertedEiPKc+0xfd) [0x9f83fb]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo15StaticShardInfo6reloadEv+0x15c) [0xb10700]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo5Shard15reloadShardInfoEv+0x13) [0xb0f377]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo17WriteBackListener3runEv+0x1be9) [0x7b2009]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x35d) [0x87d0f9]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZNK5boost4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS3_9JobStatusEEEEclEPS3_S6_+0x7d) [0x8801b7]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5boost3_bi5list2INS0_5valueIPN5mongo13BackgroundJobEEENS2_INS_10shared_ptrINS4_9JobStatusEEEEEEclINS_4_mfi3mf1IvS4_S9_EENS0_5list0EEEvNS0_4typeIvEERT_RT0_i+0x79) [0x8800b7]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5boost3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS5_9JobStatusEEEEENS0_5list2INS0_5valueIPS5_EENSB_IS8_EEEEEclEv+0x3f) [0x880035]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x1e) [0x87ffc2]
 m30999|  /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongos [0x83b1f4]
 m30999|  /lib/libpthread.so.0 [0x7f303193ea04]
 m30999|  /lib/libc.so.6(clone+0x6d) [0x7f3030efcd4d]
 m30999| Fri Mar 30 12:13:44 [WriteBackListener-localhost:30001] ERROR: backgroundjob WriteBackListener-localhost:30001error: couldn't get updated shard list from config server
 m30999| Fri Mar 30 12:13:45 [mongosMain] dbexit: received signal 15 rc:0 received signal 15
 m30999| _DEBUG warning trying to lock a mongo::mutex during static shutdown
 m30999| _DEBUG warning trying to lock a mongo::mutex during static shutdown
                3796.560049ms
Fri Mar 30 12:13:46 got signal 15 (Terminated), will terminate after current cmd ends
Fri Mar 30 12:13:46 [interruptThread] now exiting
Fri Mar 30 12:13:46 dbexit: 
Fri Mar 30 12:13:46 [interruptThread] shutdown: going to close listening sockets...
Fri Mar 30 12:13:46 [interruptThread] closing listening socket: 5
Fri Mar 30 12:13:46 [interruptThread] closing listening socket: 6
Fri Mar 30 12:13:46 [interruptThread] closing listening socket: 7
Fri Mar 30 12:13:46 [interruptThread] removing socket file: /tmp/mongodb-27999.sock
Fri Mar 30 12:13:46 [interruptThread] shutdown: going to flush diaglog...
Fri Mar 30 12:13:46 [interruptThread] shutdown: going to close sockets...
Fri Mar 30 12:13:46 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar 30 12:13:46 [interruptThread] shutdown: closing all files...
Fri Mar 30 12:13:46 [interruptThread] closeAllFiles() finished
Fri Mar 30 12:13:46 [interruptThread] shutdown: removing fs lock...
Fri Mar 30 12:13:46 dbexit: really exiting now
test /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/jstests/sharding/remove1.js exited with status 253



 Comments   
Comment by Eliot Horowitz (Inactive) [ 30/Mar/12 ]

oddly enough just commited a fix for this

Generated at Thu Feb 08 03:08:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.