[SERVER-10172] sharding/inserts_consistent.js failed on Nightly Linux 32-bit Created: 11/Jul/13  Updated: 11/Jul/16  Resolved: 12/Jul/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 2.5.1

Type: Bug Priority: Major - P3
Reporter: Matt Kangas Assignee: Andy Schwerin
Resolution: Done Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Nightly Linux 32-bit Build #1396


Issue Links:
Duplicate
is duplicated by SERVER-10196 Signal 11 during writeback_bulk_inser... Closed
is duplicated by SERVER-10173 multi_mongos1.js failed on Nightly Li... Closed
is duplicated by SERVER-10194 complex_sharding.js failed on Nightly... Closed
Operating System: ALL
Participants:

 Description   

Nightly Linux 32-bit Build #1396 July 11 (rev 8512fdda5f)

http://buildbot.mongodb.org/builders/Nightly%20Linux%2032-bit/builds/1396/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/Nightly%20Linux%2032-bit/builds/1396/test/sharding/inserts_consistent.js

----
Refreshing second mongos...
----
 m30999| Thu Jul 11 08:34:18.973 [conn1] loading chunk manager for collection inserts_consistent.coll using old chunk manager w/ version 1|0||51dea64a9f9da64575d9e961 and 1 chunks
 m30999| Thu Jul 11 08:34:18.973 [conn1] major version query from 1|0||51dea64a9f9da64575d9e961 and over 1 shards is { query: { ns: "inserts_consistent.coll", $or: [ { lastmod: { $gte: Timestamp 1000|0 } }, { shard: "shard0000", lastmod: { $gt: Timestamp 1000|0 } } ] }, orderby: { lastmod: 1 } }
 m30999| Thu Jul 11 08:34:18.974 [conn1] loaded 2 chunks into new chunk manager for inserts_consistent.coll with version 1|2||51dea64a9f9da64575d9e961
 m30999| Thu Jul 11 08:34:18.974 [conn1] ChunkManager: time to load chunks for inserts_consistent.coll: 0ms sequenceNumber: 3 version: 1|2||51dea64a9f9da64575d9e961 based on: 1|0||51dea64a9f9da64575d9e961
Thu Jul 11 08:34:19.189 TypeError: Cannot call method 'getDB' of null at /mnt/slaves/Linux_32bit_Nightly/mongo/jstests/sharding/inserts_consistent.js:21
failed to load: /mnt/slaves/Linux_32bit_Nightly/mongo/jstests/sharding/inserts_consistent.js

Greg, please triage



 Comments   
Comment by auto [ 12/Jul/13 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: SERVER-10172 Refactor RamLog to make it thread-safe.

Also prevent accidental creation of duplicate ramlogs of the same name.
Branch: master
https://github.com/mongodb/mongo/commit/6e3ccc5649d16fd9940bc8fd1b490556d5767bf8

Comment by Andy Schwerin [ 12/Jul/13 ]

Now, leaks ramlogs.

Comment by auto [ 11/Jul/13 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: SERVER-10172 Appenders must be added to global log domain during single threaded operation.

Due to an error during the logging refactoring, the RamLog("global") appender was being attached to the global log domain
during the startup of the web server thread, rather than during mongo-initializer execution. MongoS and MongoD should both have been susceptible.
Branch: master
https://github.com/mongodb/mongo/commit/30a130776b43c283fc8c38c48a9478e935a69bc2

Comment by Greg Studer [ 11/Jul/13 ]

Problem seen at:

8512fdda5f4a5d69fa9084339eec13da9de10e7a
8512fdda5f4a5d69fa9084339eec13da9de10e7a
8512fdda5f4a5d69fa9084339eec13da9de10e7a
8512fdda5f4a5d69fa9084339eec13da9de10e7a
81c207c0c5160d02dc8fe752ec662f943c0ee10d
331ca8a14dbc62cf065f3c8c6ed6ccd9a0eb9647
331ca8a14dbc62cf065f3c8c6ed6ccd9a0eb9647
86774c4091d3286b4ee2281402c67fba440e23c8

Oldest commit:
86774c4091d3286b4ee2281402c67fba440e23c8

Comment by Greg Studer [ 11/Jul/13 ]

http://buildlogs.mongodb.org/Linux%20RHEL%2064-bit/builds/1363/test/sharding/count2.js

Comment by Greg Studer [ 11/Jul/13 ]

http://buildlogs.mongodb.org/Linux%20RHEL%2064-bit/builds/1364/test/sharding/shard4.js

Comment by Greg Studer [ 11/Jul/13 ]

http://buildlogs.mongodb.org/Linux%20RHEL%2064-bit/builds/1365/test/sharding/shard2.js

Comment by Greg Studer [ 11/Jul/13 ]

Also http://buildlogs.mongodb.org/Linux%2032-bit/builds/5811/test/sharding/multi_mongos2a.js

Comment by Greg Studer [ 11/Jul/13 ]

And here:

http://buildlogs.mongodb.org/OS%20X%2010.5%2064-bit/builds/5565/test/sharding/complex_sharding.js

That's an extremely simple test, so looking at that now. Definitely something weird happens every once and awhile on js mongos startup.

Comment by Greg Studer [ 11/Jul/13 ]

Looks intermittent, another failure here:

http://buildlogs.mongodb.org/Linux%2032-bit/builds/5814/test/sharding/shard5.js

Comment by Greg Studer [ 11/Jul/13 ]

Looks like the same issue.

	
 
Thu Jul 11 08:33:12.478 shell: started program /mnt/slaves/Linux_32bit_Nightly/mongo/mongos --port 30998 --configdb localhost:29000 -vv --chunkSize 50 --setParameter enableTestCommands=1
 m30999| Thu Jul 11 08:33:12.472 [mongosMain] connection accepted from 127.0.0.1:46763 #1 (1 connection now open)
 m29000| Thu Jul 11 08:33:12.524 [initandlisten] connection accepted from 127.0.0.1:48439 #6 (6 connections now open)
 m30998| Thu Jul 11 08:33:12.490 warning: running with 1 config server should be done only for testing purposes and is not recommended for production
 m30998| Thu Jul 11 08:33:12.491 [mongosMain] MongoS version 2.5.1-pre- starting: pid=21888 port=30998 32-bit host=bs-linux32.10gen.cc (--help for usage)
 m30998| Thu Jul 11 08:33:12.491 [mongosMain] git version: 8512fdda5f4a5d69fa9084339eec13da9de10e7a
 m30998| Thu Jul 11 08:33:12.491 [mongosMain] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
 m30998| Thu Jul 11 08:33:12.491 [mongosMain] options: { chunkSize: 50, configdb: "localhost:29000", port: 30998, setParameter: [ "enableTestCommands=1" ], vv: true }
 m30998| Thu Jul 11 08:33:12.491 [mongosMain]  config string : localhost:29000
 m30998| Thu Jul 11 08:33:12.491 [mongosMain] creating new connection to:localhost:29000
 m30998| Thu Jul 11 08:33:12.491 BackgroundJob starting: ConnectBG
 m30998| Thu Jul 11 08:33:12.491 [mongosMain] connected connection!
 m30998| Thu Jul 11 08:33:12.530 creating new connection to:localhost:29000
 m30998| Thu Jul 11 08:33:12.530 BackgroundJob starting: ConnectBG
 m30998| Thu Jul 11 08:33:12.531 [mongosMain] MaxChunkSize: 50
 m30998| Thu Jul 11 08:33:12.533 [mongosMain] fd limit hard:1024 soft:1024 max conn: 819
 m30998| Thu Jul 11 08:33:12.533 [mongosMain] waiting for connections on port 30998
 m30998| Thu Jul 11 08:33:12.533 BackgroundJob starting: Balancer
 m30998| Thu Jul 11 08:33:12.533 [Balancer] about to contact config servers and shards
 m30998| Thu Jul 11 08:33:12.533 BackgroundJob starting: cursorTimeout
 m30998| Thu Jul 11 08:33:12.534 BackgroundJob starting: PeriodicTask::Runner
 m30998| Thu Jul 11 08:33:12.534 [Balancer] config servers and shards contacted successfully
 m30998| Thu Jul 11 08:33:12.534 [Balancer] balancer id: bs-linux32.10gen.cc:30998 started at Jul 11 08:33:12
 m30998| Thu Jul 11 08:33:12.534 [Balancer] created new distributed lock for balancer on localhost:29000 ( lock timeout : 900000, ping interval : 30000, process : 0 )
 m30998| Thu Jul 11 08:33:12.534 [Balancer] creating new connection to:localhost:29000
 m29000| Thu Jul 11 08:33:12.576 [conn6] end connection 127.0.0.1:48439 (5 connections now open)
Could not start mongo program at 30998, process ended

It's not clear if mongos is crashing on startup or something?

Generated at Thu Feb 08 03:22:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.