[SERVER-10618] complex_sharding.js HANGING on Windows 32-bit only Created: 24/Aug/13  Updated: 11/Jul/16  Resolved: 28/Aug/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 2.5.2
Fix Version/s: 2.5.3

Type: Bug Priority: Major - P3
Reporter: Matt Kangas Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Ugh ugh ugh.

The Buildbot Windows 32-bit builders (master and nightly) are HANGING on sharding/complex_sharding.js

Here is one example:

Nightly Windows 32-bit Build #1402 Friday Aug 23 (rev b283169ae51)

http://buildbot.10gen.cc/builders/Nightly%20Windows%2032-bit/builds/1402/steps/mongosTest_smokeSharding/logs/stdio

 *******************************************
         Test : complex_sharding.js ...
      Command : ...
         Date : Fri Aug 23 18:04:09 2013
                (output suppressed; see http://buildlogs.mongodb.org/build/5217706ad2a60f2040000824/test/5217a41ad2a60f3124000158/)
...
command timed out: 10800 seconds without output, attempting to kill
                  345.9525 minutes

The log saved to buildlogger is huge. For that specific build, `http://buildlogs.mongodb.org/Nightly%20Windows%2032-bit/builds/1402/test/sharding/complex_sharding.js?mode=raw` is 286MB.

Most of this file consists of this line repeated...

 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1

That line appears 3,372,664 times, right unto the very end.

This is not impacting MCI because it has no 32-bit Windows builders.

This morning, I noticed that the nightly Win32 builder had not run by 9am because the master builder was still running. In retrospect, it was hanging on complex_sharding.js. Then the nightly builder hung in precisely the same way.



 Comments   
Comment by Greg Studer [ 28/Aug/13 ]

Fixed by other issues, test rename helpful for future problems.

Comment by auto [ 28/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-10618 complex_sharding.js -> basic_sharding.js invert verbosity for less verbose rs nodes
Branch: master
https://github.com/mongodb/mongo/commit/31f032eacc98c0dae39edaedb5bcee25f4bf23b5

Comment by Matt Dannenberg [ 24/Aug/13 ]

matt.kangas@10gen.com was actually the one who figured out the sync_passive2.js failure. But from how greg_10gen describes the test, it seems entirely plausible that this is also a hardware/windows load balancer caused failure. Since we seem to have multiple test failures caused by this, it may be worth setting up a multicored windows 32bit buildslave. But I don't know how much work that takes, so maybe not.

That said, if when I examine the sync_passive2.js failure I am able to solve it, it may also fix this one. Unless the loop and logging greg_10gen mentions is from sharding and is the cause of the problem.

Comment by Greg Studer [ 24/Aug/13 ]

I don't know enough about replica sets to tell, but could this be caused by the same problem as sync_passive2.js? - one of the servers loops forever b/c no progress is made by another server on a single-core system. mattd@10gen.com may know more, looks like he's recently looked at these files.

Comment by Greg Studer [ 24/Aug/13 ]

This test, despite the name, does basically no work and it's only job is to ensure that new-style parameters to ShardingTest are correctly applied.

However, it does start servers at very high verbosity - maybe there's an extremely inefficient loop that gets made worse by logging.

Comment by Matt Kangas [ 24/Aug/13 ]

Known builds that have hung on complex_sharding.js

Windows 32-bit master:

Nightly Windows 32-bit:

THAT'S IT. NO OTHER EXAMPLES IN BUILDBOT HISTORY.

However, Win 32 builders have been failing for so long on sync_passive2.js (SERVER-10341) that it's possible this test has not run at all since the last green run.

Last green runs on Win 32 builders:

Comment by Matt Kangas [ 24/Aug/13 ]

First 400 lines of the 286 MB log for `http://buildlogs.mongodb.org/Nightly%20Windows%2032-bit/builds/1402/test/sharding/complex_sharding.js?mode=raw`

 
MongoDB shell version: 2.5.2
MONGODB-CR
Resetting db path '/data/db/test0'
2013-08-23T18:04:10.356+0000 shell: started program mongod.exe --port 30000 --dbpath /data/db/test0 -vvvv --setParameter enableTestCommands=1
 m30000| 2013-08-23T18:04:10.371+0000 
 m30000| 2013-08-23T18:04:10.371+0000 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
 m30000| 2013-08-23T18:04:10.371+0000 
 m30000| 2013-08-23T18:04:10.387+0000 shardObjTest passed
 m30000| 2013-08-23T18:04:10.387+0000 isInRangeTest passed
 m30000| 2013-08-23T18:04:10.387+0000 shardKeyTest passed
 m30000| 2013-08-23T18:04:10.387+0000 versionArrayTest passed
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] MongoDB starting : pid=7700 port=30000 dbpath=/data/db/test0 32-bit host=EC2AMAZ-EXWX6MN
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] 
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] ** NOTE: This is a development version (2.5.2) of MongoDB.
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] **       Not recommended for production.
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] 
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] 
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] db version v2.5.2
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] git version: b283169ae51328f3fb5e3c8a3c9017a2e42c148f
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] allocator: system
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] options: { dbpath: "/data/db/test0", port: 30000, setParameter: [ "enableTestCommands=1" ], vvvv: true }
 m30000| 2013-08-23T18:04:10.387+0000 BackgroundJob starting: DataFileSync
 m30000| 2013-08-23T18:04:10.387+0000 [initandlisten] opening db:  local
 m30000| 2013-08-23T18:04:10.403+0000 [initandlisten] enter repairDatabases (to check pdfile version #)
 m30000| 2013-08-23T18:04:10.403+0000 [initandlisten] done repairDatabases
 m30000| 2013-08-23T18:04:10.403+0000 [initandlisten] runQuery called local.$cmd { create: "startup_log", size: 10485760, capped: true }
 m30000| 2013-08-23T18:04:10.403+0000 [initandlisten] run command local.$cmd { create: "startup_log", size: 10485760, capped: true }
 m30000| 2013-08-23T18:04:10.403+0000 [initandlisten] create collection local.startup_log { create: "startup_log", size: 10485760, capped: true }
 m30000| 2013-08-23T18:04:10.403+0000 [initandlisten] mmf create /data/db/test0\local.0
 m30000| 2013-08-23T18:04:10.403+0000 [FileAllocator] allocating new datafile /data/db/test0\local.0, filling with zeroes...
 m30000| 2013-08-23T18:04:10.403+0000 [FileAllocator] creating directory /data/db/test0\_tmp
 m30000| 2013-08-23T18:04:10.449+0000 [FileAllocator] done allocating datafile /data/db/test0\local.0, size: 16MB,  took 0.047 secs
 m30000| 2013-08-23T18:04:10.449+0000 [initandlisten] mmf finishOpening 06490000 D:/data/db/test0/local.0 len:16777216
 m30000| 2013-08-23T18:04:10.449+0000 [initandlisten] mmf create /data/db/test0\local.ns
 m30000| 2013-08-23T18:04:10.449+0000 [FileAllocator] allocating new datafile /data/db/test0\local.ns, filling with zeroes...
 m30000| 2013-08-23T18:04:10.496+0000 [FileAllocator] done allocating datafile /data/db/test0\local.ns, size: 16MB,  took 0.047 secs
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] mmf finishOpening 07490000 D:/data/db/test0/local.ns len:16777216
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] ExtentManager: creating new extent for: local.startup_log in file: 0 size: 10485760
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] allocExtent local.startup_log size 10485760 0
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] New namespace: local.startup_log
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] ExtentManager: creating new extent for: local.system.namespaces in file: 0 size: 8192
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] allocExtent local.system.namespaces size 8192 0
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] New namespace: local.system.namespaces
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37 101ms
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] insert local.startup_log ninserted:1 keyUpdates:0  0ms
 m30000| 2013-08-23T18:04:10.496+0000 [initandlisten] waiting for connections on port 30000
 m30000| 2013-08-23T18:04:10.496+0000 BackgroundJob starting: snapshot
 m30000| 2013-08-23T18:04:10.496+0000 BackgroundJob starting: ClientCursorMonitor
 m30000| 2013-08-23T18:04:10.496+0000 BackgroundJob starting: PeriodicTask::Runner
 m30000| 2013-08-23T18:04:10.496+0000 BackgroundJob starting: TTLMonitor
 m30000| 2013-08-23T18:04:10.496+0000 BackgroundJob starting: IndexRebuilder
 m30000| 2013-08-23T18:04:10.496+0000 [IndexRebuilder] runQuery called local.system.namespaces {}
 m30000| 2013-08-23T18:04:10.496+0000 [IndexRebuilder] query local.system.namespaces ntoreturn:0 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:162 nreturned:1 reslen:114 0ms
 m30000| 2013-08-23T18:04:10.496+0000 [IndexRebuilder] checking ns local.startup_log for interrupted index builds
 m30000| 2013-08-23T18:04:10.496+0000 [IndexRebuilder] checking complete
 m30000| 2013-08-23T18:04:10.886+0000 [initandlisten] connection accepted from 127.0.0.1:64856 #1 (1 connection now open)
Replica set test!
ReplSetTest Starting Set
ReplSetTest n is : 0
ReplSetTest n: 0 ports: [ 31200, 31201 ]	31200 number
{
	"useHostName" : true,
	"oplogSize" : 40,
	"keyFile" : undefined,
	"port" : 31200,
	"noprealloc" : "",
	"smallfiles" : "",
	"rest" : "",
	"replSet" : "test-rs1",
	"dbpath" : "$set-$node",
	"useHostname" : true,
	"noJournalPrealloc" : undefined,
	"pathOpts" : {
		"testName" : "test",
		"shard" : 1,
		"node" : 0,
		"set" : "test-rs1"
	},
	"verbose" : 5,
	"restart" : undefined
}
ReplSetTest Starting....
Resetting db path '/data/db/test-rs1-0'
2013-08-23T18:04:10.902+0000 shell: started program mongod.exe --oplogSize 40 --port 31200 --noprealloc --smallfiles --rest --replSet test-rs1 --dbpath /data/db/test-rs1-0 -vvvvv --setParameter enableTestCommands=1
 m31200| 2013-08-23T18:04:10.933+0000 ** WARNING: --rest is specified without --httpinterface,
 m31200| 2013-08-23T18:04:10.933+0000 **          enabling http interface
 m31200| note: noprealloc may hurt performance in many applications
 m31200| 2013-08-23T18:04:10.933+0000 
 m31200| 2013-08-23T18:04:10.933+0000 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
 m31200| 2013-08-23T18:04:10.933+0000 
 m31200| 2013-08-23T18:04:10.933+0000 shardObjTest passed
 m31200| 2013-08-23T18:04:10.933+0000 isInRangeTest passed
 m31200| 2013-08-23T18:04:10.933+0000 shardKeyTest passed
 m31200| 2013-08-23T18:04:10.933+0000 versionArrayTest passed
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] MongoDB starting : pid=1784 port=31200 dbpath=/data/db/test-rs1-0 32-bit host=EC2AMAZ-EXWX6MN
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] 
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] ** NOTE: This is a development version (2.5.2) of MongoDB.
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] **       Not recommended for production.
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] 
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] 
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] db version v2.5.2
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] git version: b283169ae51328f3fb5e3c8a3c9017a2e42c148f
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] allocator: system
 m31200| 2013-08-23T18:04:10.933+0000 [initandlisten] options: { dbpath: "/data/db/test-rs1-0", noprealloc: true, oplogSize: 40, port: 31200, replSet: "test-rs1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true, vvvvv: true }
 m31200| 2013-08-23T18:04:10.933+0000 BackgroundJob starting: DataFileSync
 m31200| 2013-08-23T18:04:10.949+0000 [initandlisten] enter repairDatabases (to check pdfile version #)
 m31200| 2013-08-23T18:04:10.949+0000 [initandlisten] done repairDatabases
 m31200| 2013-08-23T18:04:10.949+0000 [initandlisten] runQuery called local.$cmd { create: "startup_log", size: 10485760, capped: true }
 m31200| 2013-08-23T18:04:10.949+0000 [initandlisten] run command local.$cmd { create: "startup_log", size: 10485760, capped: true }
 m31200| 2013-08-23T18:04:10.964+0000 [initandlisten] opening db:  local
 m31200| 2013-08-23T18:04:10.964+0000 [initandlisten] create collection local.startup_log { create: "startup_log", size: 10485760, capped: true }
 m31200| 2013-08-23T18:04:10.964+0000 [initandlisten] mmf create /data/db/test-rs1-0\local.0
 m31200| 2013-08-23T18:04:10.964+0000 [FileAllocator] allocating new datafile /data/db/test-rs1-0\local.0, filling with zeroes...
 m31200| 2013-08-23T18:04:10.964+0000 [FileAllocator] creating directory /data/db/test-rs1-0\_tmp
 m31200| 2013-08-23T18:04:11.011+0000 [FileAllocator] done allocating datafile /data/db/test-rs1-0\local.0, size: 16MB,  took 0.047 secs
 m31200| 2013-08-23T18:04:11.011+0000 [initandlisten] mmf finishOpening 06930000 D:/data/db/test-rs1-0/local.0 len:16777216
 m31200| 2013-08-23T18:04:11.011+0000 [initandlisten] mmf create /data/db/test-rs1-0\local.ns
 m31200| 2013-08-23T18:04:11.011+0000 [FileAllocator] allocating new datafile /data/db/test-rs1-0\local.ns, filling with zeroes...
 m31200| 2013-08-23T18:04:11.058+0000 [FileAllocator] done allocating datafile /data/db/test-rs1-0\local.ns, size: 16MB,  took 0.047 secs
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] mmf finishOpening 07930000 D:/data/db/test-rs1-0/local.ns len:16777216
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] ExtentManager: creating new extent for: local.startup_log in file: 0 size: 10485760
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] allocExtent local.startup_log size 10485760 0
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] New namespace: local.startup_log
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] ExtentManager: creating new extent for: local.system.namespaces in file: 0 size: 8192
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] allocExtent local.system.namespaces size 8192 0
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] New namespace: local.system.namespaces
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37 101ms
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] insert local.startup_log ninserted:1 keyUpdates:0  0ms
 m31200| 2013-08-23T18:04:11.058+0000 [initandlisten] waiting for connections on port 31200
 m31200| 2013-08-23T18:04:11.058+0000 BackgroundJob starting: snapshot
 m31200| 2013-08-23T18:04:11.058+0000 BackgroundJob starting: ClientCursorMonitor
 m31200| 2013-08-23T18:04:11.058+0000 BackgroundJob starting: PeriodicTask::Runner
 m31200| 2013-08-23T18:04:11.058+0000 BackgroundJob starting: TTLMonitor
 m31200| 2013-08-23T18:04:11.058+0000 BackgroundJob starting: IndexRebuilder
 m31200| 2013-08-23T18:04:11.058+0000 [IndexRebuilder] runQuery called local.system.namespaces {}
 m31200| 2013-08-23T18:04:11.058+0000 [IndexRebuilder] query local.system.namespaces ntoreturn:0 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:161 nreturned:1 reslen:114 0ms
 m31200| 2013-08-23T18:04:11.058+0000 [IndexRebuilder] checking ns local.startup_log for interrupted index builds
 m31200| 2013-08-23T18:04:11.058+0000 [IndexRebuilder] checking complete
 m31200| 2013-08-23T18:04:11.058+0000 [rsStart] replSet beginning startup...
 m31200| 2013-08-23T18:04:11.058+0000 [rsStart] loadConfig() local.system.replset
 m31200| 2013-08-23T18:04:11.058+0000 [rsStart] runQuery called local.system.replset {}
 m31200| 2013-08-23T18:04:11.058+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:177 nreturned:0 reslen:20 0ms
 m31200| 2013-08-23T18:04:11.058+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31200| 2013-08-23T18:04:11.058+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
 m31200| 2013-08-23T18:04:11.058+0000 [rsStart] replSet info no seed hosts were specified on the --replSet command line
 m31200| 2013-08-23T18:04:11.058+0000 [websvr] admin web console waiting for connections on port 32200
 m31200| 2013-08-23T18:04:11.432+0000 [initandlisten] connection accepted from 127.0.0.1:64857 #1 (1 connection now open)
[ connection to EC2AMAZ-EXWX6MN:31200 ]
ReplSetTest n is : 1
ReplSetTest n: 1 ports: [ 31200, 31201 ]	31201 number
{
	"useHostName" : true,
	"oplogSize" : 40,
	"keyFile" : undefined,
	"port" : 31201,
	"noprealloc" : "",
	"smallfiles" : "",
	"rest" : "",
	"replSet" : "test-rs1",
	"dbpath" : "$set-$node",
	"useHostname" : true,
	"noJournalPrealloc" : undefined,
	"pathOpts" : {
		"testName" : "test",
		"shard" : 1,
		"node" : 1,
		"set" : "test-rs1"
	},
	"verbose" : 6,
	"arbiter" : true,
	"restart" : undefined
}
ReplSetTest Starting....
Resetting db path '/data/db/test-rs1-1'
2013-08-23T18:04:11.432+0000 shell: started program mongod.exe --oplogSize 1 --port 31201 --noprealloc --smallfiles --rest --replSet test-rs1 --dbpath /data/db/test-rs1-1 -vvvvvv --setParameter enableTestCommands=1
 m31201| 2013-08-23T18:04:11.463+0000 ** WARNING: --rest is specified without --httpinterface,
 m31201| 2013-08-23T18:04:11.463+0000 **          enabling http interface
 m31201| note: noprealloc may hurt performance in many applications
 m31201| 2013-08-23T18:04:11.463+0000 
 m31201| 2013-08-23T18:04:11.463+0000 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
 m31201| 2013-08-23T18:04:11.463+0000 
 m31201| 2013-08-23T18:04:11.463+0000 shardObjTest passed
 m31201| 2013-08-23T18:04:11.463+0000 isInRangeTest passed
 m31201| 2013-08-23T18:04:11.463+0000 shardKeyTest passed
 m31201| 2013-08-23T18:04:11.463+0000 versionArrayTest passed
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] MongoDB starting : pid=5956 port=31201 dbpath=/data/db/test-rs1-1 32-bit host=EC2AMAZ-EXWX6MN
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] 
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] ** NOTE: This is a development version (2.5.2) of MongoDB.
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] **       Not recommended for production.
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] 
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] 
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] db version v2.5.2
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] git version: b283169ae51328f3fb5e3c8a3c9017a2e42c148f
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] allocator: system
 m31201| 2013-08-23T18:04:11.463+0000 [initandlisten] options: { dbpath: "/data/db/test-rs1-1", noprealloc: true, oplogSize: 1, port: 31201, replSet: "test-rs1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true, vvvvvv: true }
 m31201| 2013-08-23T18:04:11.463+0000 BackgroundJob starting: DataFileSync
 m31201| 2013-08-23T18:04:11.479+0000 [initandlisten] enter repairDatabases (to check pdfile version #)
 m31201| 2013-08-23T18:04:11.479+0000 [initandlisten] done repairDatabases
 m31201| 2013-08-23T18:04:11.479+0000 [initandlisten] runQuery called local.$cmd { create: "startup_log", size: 10485760, capped: true }
 m31201| 2013-08-23T18:04:11.479+0000 [initandlisten] run command local.$cmd { create: "startup_log", size: 10485760, capped: true }
 m31201| 2013-08-23T18:04:11.495+0000 [initandlisten] opening db:  local
 m31201| 2013-08-23T18:04:11.495+0000 [initandlisten] create collection local.startup_log { create: "startup_log", size: 10485760, capped: true }
 m31201| 2013-08-23T18:04:11.495+0000 [initandlisten] mmf create /data/db/test-rs1-1\local.0
 m31201| 2013-08-23T18:04:11.495+0000 [FileAllocator] allocating new datafile /data/db/test-rs1-1\local.0, filling with zeroes...
 m31201| 2013-08-23T18:04:11.495+0000 [FileAllocator] creating directory /data/db/test-rs1-1\_tmp
 m31201| 2013-08-23T18:04:11.541+0000 [FileAllocator] done allocating datafile /data/db/test-rs1-1\local.0, size: 16MB,  took 0.047 secs
 m31201| 2013-08-23T18:04:11.541+0000 [initandlisten] mmf finishOpening 06310000 D:/data/db/test-rs1-1/local.0 len:16777216
 m31201| 2013-08-23T18:04:11.541+0000 [initandlisten] mmf create /data/db/test-rs1-1\local.ns
 m31201| 2013-08-23T18:04:11.541+0000 [FileAllocator] allocating new datafile /data/db/test-rs1-1\local.ns, filling with zeroes...
 m31201| 2013-08-23T18:04:11.588+0000 [FileAllocator] done allocating datafile /data/db/test-rs1-1\local.ns, size: 16MB,  took 0.048 secs
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] mmf finishOpening 07310000 D:/data/db/test-rs1-1/local.ns len:16777216
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] ExtentManager: creating new extent for: local.startup_log in file: 0 size: 10485760
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] allocExtent local.startup_log size 10485760 0
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] New namespace: local.startup_log
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] ExtentManager: creating new extent for: local.system.namespaces in file: 0 size: 8192
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] allocExtent local.system.namespaces size 8192 0
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] New namespace: local.system.namespaces
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37 102ms
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] insert local.startup_log ninserted:1 keyUpdates:0  0ms
 m31201| 2013-08-23T18:04:11.588+0000 [initandlisten] waiting for connections on port 31201
 m31201| 2013-08-23T18:04:11.588+0000 BackgroundJob starting: snapshot
 m31201| 2013-08-23T18:04:11.588+0000 BackgroundJob starting: ClientCursorMonitor
 m31201| 2013-08-23T18:04:11.588+0000 BackgroundJob starting: PeriodicTask::Runner
 m31201| 2013-08-23T18:04:11.588+0000 BackgroundJob starting: TTLMonitor
 m31201| 2013-08-23T18:04:11.588+0000 BackgroundJob starting: IndexRebuilder
 m31201| 2013-08-23T18:04:11.588+0000 [IndexRebuilder] runQuery called local.system.namespaces {}
 m31201| 2013-08-23T18:04:11.588+0000 [IndexRebuilder] query local.system.namespaces ntoreturn:0 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:175 nreturned:1 reslen:114 0ms
 m31201| 2013-08-23T18:04:11.588+0000 [IndexRebuilder] checking ns local.startup_log for interrupted index builds
 m31201| 2013-08-23T18:04:11.588+0000 [IndexRebuilder] checking complete
 m31201| 2013-08-23T18:04:11.588+0000 [rsStart] replSet beginning startup...
 m31201| 2013-08-23T18:04:11.588+0000 [rsStart] loadConfig() local.system.replset
 m31201| 2013-08-23T18:04:11.588+0000 [rsStart] runQuery called local.system.replset {}
 m31201| 2013-08-23T18:04:11.588+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:200 nreturned:0 reslen:20 0ms
 m31201| 2013-08-23T18:04:11.588+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31201| 2013-08-23T18:04:11.588+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
 m31201| 2013-08-23T18:04:11.604+0000 [rsStart] replSet info no seed hosts were specified on the --replSet command line
 m31201| 2013-08-23T18:04:11.604+0000 [websvr] admin web console waiting for connections on port 32201
 m31201| 2013-08-23T18:04:11.963+0000 [initandlisten] connection accepted from 127.0.0.1:64858 #1 (1 connection now open)
[ connection to EC2AMAZ-EXWX6MN:31200, connection to EC2AMAZ-EXWX6MN:31201 ]
{
	"replSetInitiate" : {
		"_id" : "test-rs1",
		"members" : [
			{
				"_id" : 0,
				"host" : "EC2AMAZ-EXWX6MN:31200"
			},
			{
				"_id" : 1,
				"host" : "EC2AMAZ-EXWX6MN:31201",
				"arbiterOnly" : true
			}
		]
	}
}
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] runQuery called admin.$cmd { replSetInitiate: { _id: "test-rs1", members: [ { _id: 0.0, host: "EC2AMAZ-EXWX6MN:31200" }, { _id: 1.0, host: "EC2AMAZ-EXWX6MN:31201", arbiterOnly: true } ] } }
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] run command admin.$cmd { replSetInitiate: { _id: "test-rs1", members: [ { _id: 0.0, host: "EC2AMAZ-EXWX6MN:31200" }, { _id: 1.0, host: "EC2AMAZ-EXWX6MN:31201", arbiterOnly: true } ] } }
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] command: { replSetInitiate: { _id: "test-rs1", members: [ { _id: 0.0, host: "EC2AMAZ-EXWX6MN:31200" }, { _id: 1.0, host: "EC2AMAZ-EXWX6MN:31201", arbiterOnly: true } ] } }
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] replSet replSetInitiate admin command received from client
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] replSet replSetInitiate config object parses ok, 2 members specified
 m31200| 2013-08-23T18:04:11.963+0000 BackgroundJob starting: ConnectBG
 m31200| 2013-08-23T18:04:11.963+0000 [initandlisten] connection accepted from 10.151.17.233:64861 #2 (2 connections now open)
 m31200| 2013-08-23T18:04:11.963+0000 [conn2] runQuery called admin.$cmd { _isSelf: 1 }
 m31200| 2013-08-23T18:04:11.963+0000 [conn2] run command admin.$cmd { _isSelf: 1 }
 m31200| 2013-08-23T18:04:11.963+0000 [conn2] command admin.$cmd command: { _isSelf: 1 } ntoreturn:1 keyUpdates:0  reslen:53 0ms
 m31200| 2013-08-23T18:04:11.963+0000 [conn2] Socket recv() conn closed? 10.151.17.233:64861
 m31200| 2013-08-23T18:04:11.963+0000 [conn2] SocketException: remote: 10.151.17.233:64861 error: 9001 socket exception [CLOSED] server [10.151.17.233:64861] 
 m31200| 2013-08-23T18:04:11.963+0000 [conn2] end connection 10.151.17.233:64861 (1 connection now open)
 m31200| 2013-08-23T18:04:11.963+0000 BackgroundJob starting: ConnectBG
 m31201| 2013-08-23T18:04:11.963+0000 [initandlisten] connection accepted from 10.151.17.233:64862 #2 (2 connections now open)
 m31201| 2013-08-23T18:04:11.963+0000 [conn2] runQuery called admin.$cmd { replSetHeartbeat: "test-rs1", v: -1, pv: 1, checkEmpty: true, from: "" }
 m31201| 2013-08-23T18:04:11.963+0000 [conn2] run command admin.$cmd { replSetHeartbeat: "test-rs1", v: -1, pv: 1, checkEmpty: true, from: "" }
 m31201| 2013-08-23T18:04:11.963+0000 [conn2] command: { replSetHeartbeat: "test-rs1", v: -1, pv: 1, checkEmpty: true, from: "" }
 m31201| 2013-08-23T18:04:11.963+0000 [conn2] command admin.$cmd command: { replSetHeartbeat: "test-rs1", v: -1, pv: 1, checkEmpty: true, from: "" } ntoreturn:1 keyUpdates:0 locks(micros) r:152 reslen:82 0ms
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] replSet replSetInitiate all members seem up
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] ******
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] creating replication oplog of size: 40MB...
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] create collection local.oplog.rs { size: 41943040.0, capped: true, autoIndexId: false }
 m31200| 2013-08-23T18:04:11.963+0000 [conn1] mmf create /data/db/test-rs1-0\local.1
 m31200| 2013-08-23T18:04:11.963+0000 [FileAllocator] allocating new datafile /data/db/test-rs1-0\local.1, filling with zeroes...
 m31200| 2013-08-23T18:04:12.165+0000 [rsStart] runQuery called local.system.replset {}
 m31200| 2013-08-23T18:04:12.165+0000 [FileAllocator] done allocating datafile /data/db/test-rs1-0\local.1, size: 64MB,  took 0.193 secs
 m31200| 2013-08-23T18:04:12.165+0000 [conn1] mmf finishOpening 090D0000 D:/data/db/test-rs1-0/local.1 len:67108864
 m31200| 2013-08-23T18:04:12.165+0000 [conn1] ExtentManager: creating new extent for: local.oplog.rs in file: 1 size: 41943040
 m31200| 2013-08-23T18:04:12.165+0000 [conn1] allocExtent local.oplog.rs size 41943040 0
 m31200| 2013-08-23T18:04:12.165+0000 [conn1] New namespace: local.oplog.rs
 m31201| 2013-08-23T18:04:12.618+0000 [rsStart] runQuery called local.system.replset {}
 m31201| 2013-08-23T18:04:12.618+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:122 nreturned:0 reslen:20 0ms
 m31201| 2013-08-23T18:04:12.618+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31201| 2013-08-23T18:04:12.618+0000 [rsStart] replSet info no seed hosts were specified on the --replSet command line
 m31201| 2013-08-23T18:04:13.632+0000 [rsStart] runQuery called local.system.replset {}
 m31201| 2013-08-23T18:04:13.632+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:171 nreturned:0 reslen:20 0ms
 m31201| 2013-08-23T18:04:13.632+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31201| 2013-08-23T18:04:13.632+0000 [rsStart] replSet info no seed hosts were specified on the --replSet command line
 m31201| 2013-08-23T18:04:14.646+0000 [rsStart] runQuery called local.system.replset {}
 m31201| 2013-08-23T18:04:14.646+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:121 nreturned:0 reslen:20 0ms
 m31201| 2013-08-23T18:04:14.646+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31201| 2013-08-23T18:04:14.646+0000 [rsStart] replSet info no seed hosts were specified on the --replSet command line
 m31201| 2013-08-23T18:04:15.660+0000 [rsStart] runQuery called local.system.replset {}
 m31201| 2013-08-23T18:04:15.660+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:148 nreturned:0 reslen:20 0ms
 m31201| 2013-08-23T18:04:15.660+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31201| 2013-08-23T18:04:15.660+0000 [rsStart] replSet info no seed hosts were specified on the --replSet command line
 m31200| 2013-08-23T18:04:16.299+0000 [conn1] ******
 m31200| 2013-08-23T18:04:16.299+0000 [conn1] replSet info saving a newer config version to local.system.replset
 m31201| 2013-08-23T18:04:16.674+0000 [rsStart] runQuery called local.system.replset {}
 m31201| 2013-08-23T18:04:16.674+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:129 nreturned:0 reslen:20 0ms
 m31201| 2013-08-23T18:04:16.674+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31201| 2013-08-23T18:04:16.674+0000 [rsStart] replSet info no seed hosts were specified on the --replSet command line
 m31200| 2013-08-23T18:04:16.721+0000 [conn1] ExtentManager: creating new extent for: local.system.replset in file: 1 size: 12288
 m31200| 2013-08-23T18:04:16.721+0000 [conn1] allocExtent local.system.replset size 12288 0
 m31200| 2013-08-23T18:04:16.721+0000 [conn1] New namespace: local.system.replset
 m31200| 2013-08-23T18:04:16.721+0000 [conn1] replSet saveConfigLocally done
 m31200| 2013-08-23T18:04:16.721+0000 [conn1] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
 m31200| 2013-08-23T18:04:16.721+0000 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "test-rs1", members: [ { _id: 0.0, host: "EC2AMAZ-EXWX6MN:31200" }, { _id: 1.0, host: "EC2AMAZ-EXWX6MN:31201", arbiterOnly: true } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:4759955 reslen:112 4766ms
{
	"info" : "Config now saved locally.  Should come online in about a minute.",
	"ok" : 1
}
 m31200| 2013-08-23T18:04:16.736+0000 [conn1] runQuery called admin.$cmd { ismaster: 1.0 }
 m31200| 2013-08-23T18:04:16.736+0000 [conn1] run command admin.$cmd { ismaster: 1.0 }
 m31200| 2013-08-23T18:04:16.736+0000 [conn1] command admin.$cmd command: { ismaster: 1.0 } ntoreturn:1 keyUpdates:0  reslen:247 0ms
 m31201| 2013-08-23T18:04:16.736+0000 [conn1] runQuery called admin.$cmd { ismaster: 1.0 }
 m31201| 2013-08-23T18:04:16.736+0000 [conn1] run command admin.$cmd { ismaster: 1.0 }
 m31201| 2013-08-23T18:04:16.736+0000 [conn1] command admin.$cmd command: { ismaster: 1.0 } ntoreturn:1 keyUpdates:0  reslen:266 0ms
 m31200| 2013-08-23T18:04:16.736+0000 [rsStart] query local.system.replset ntoreturn:1 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:88 nreturned:1 reslen:182 0ms
 m31200| 2013-08-23T18:04:16.736+0000 [rsStart] replSet I am EC2AMAZ-EXWX6MN:31200
 m31200| 2013-08-23T18:04:16.736+0000 [rsStart] replSet STARTUP2
 m31200| 2013-08-23T18:04:16.736+0000 BackgroundJob starting: rsHealthPoll
 m31201| 2013-08-23T18:04:16.736+0000 [conn2] runQuery called admin.$cmd { replSetHeartbeat: "test-rs1", v: 1, pv: 1, checkEmpty: false, from: "EC2AMAZ-EXWX6MN:31200" }
 m31201| 2013-08-23T18:04:16.736+0000 [conn2] run command admin.$cmd { replSetHeartbeat: "test-rs1", v: 1, pv: 1, checkEmpty: false, from: "EC2AMAZ-EXWX6MN:31200" }
 m31201| 2013-08-23T18:04:16.736+0000 [conn2] command: { replSetHeartbeat: "test-rs1", v: 1, pv: 1, checkEmpty: false, from: "EC2AMAZ-EXWX6MN:31200" }
 m31201| 2013-08-23T18:04:16.736+0000 [conn2] command admin.$cmd command: { replSetHeartbeat: "test-rs1", v: 1, pv: 1, checkEmpty: false, from: "EC2AMAZ-EXWX6MN:31200" } ntoreturn:1 keyUpdates:0  reslen:72 0ms
 m31200| 2013-08-23T18:04:16.736+0000 [rsHealthPoll] replSet member EC2AMAZ-EXWX6MN:31201 is up
 m31200| 2013-08-23T18:04:16.736+0000 BackgroundJob starting: rsMgr
 m31200| 2013-08-23T18:04:16.736+0000 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
 m31200| 2013-08-23T18:04:16.736+0000 [rsSync] replSet SECONDARY
 m31200| 2013-08-23T18:04:16.736+0000 [rsBackgroundSync] replset bgsync fetch queue set to: 5217a420:1 0
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] ExtentManager: creating new extent for: local.me in file: 1 size: 4096
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] allocExtent local.me size 4096 0
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] adding _id index for collection local.me
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] ExtentManager: creating new extent for: local.system.indexes in file: 1 size: 4096
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] allocExtent local.system.indexes size 4096 0
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] New namespace: local.system.indexes
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] build index local.me { _id: 1 }
 m31200| mem info: before index start vsize: 213 resident: 40 mapped: 96
 m31200| mem info: before final sort vsize: 213 resident: 40 mapped: 96
 m31200| mem info: after final sort vsize: 213 resident: 40 mapped: 96
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] 	 external sort used : 0 files  in 0 secs
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] ExtentManager: creating new extent for: local.me.$_id_ in file: 1 size: 36864
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] allocExtent local.me.$_id_ size 36864 0
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] New namespace: local.me.$_id_
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] 	 done building bottom layer, going to commit
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] build index done.  scanned 0 total records. 0.001 secs
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] New namespace: local.me
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1
 m31200| 2013-08-23T18:04:16.736+0000 [rsSyncNotifier] replset markOplog: 0:0 5217a420:1

Generated at Thu Feb 08 03:23:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.