MongoDB shell version: 3.2.0-rc2 ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 20000, 20001, 20002 ] 20000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 20000, "noprealloc" : "", "smallfiles" : "", "replSet" : "dbhash", "dbpath" : "$set-$node", "storageEngine" : "wiredTiger", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "dbhash" } } ReplSetTest Starting.... Resetting db path '/data/db/dbhash-0' 2015-11-04T16:50:59.555-0500 I - [thread1] shell: started program (sh18621): mongod --oplogSize 40 --port 20000 --noprealloc --smallfiles --replSet dbhash --dbpath /data/db/dbhash-0 --storageEngine wiredTiger --setParameter enableTestCommands=1 2015-11-04T16:50:59.555-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20000, reason: errno:111 Connection refused d20000| note: noprealloc may hurt performance in many applications d20000| 2015-11-04T16:50:59.581-0500 I CONTROL [initandlisten] MongoDB starting : pid=18621 port=20000 dbpath=/data/db/dbhash-0 64-bit host=CAP-JA-Ubuntu14 d20000| 2015-11-04T16:50:59.581-0500 I CONTROL [initandlisten] db version v3.2.0-rc2 d20000| 2015-11-04T16:50:59.581-0500 I CONTROL [initandlisten] git version: 8a3acb42742182c5e314636041c2df368232bbc5 d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014 d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] allocator: tcmalloc d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] modules: none d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] build environment: d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] distmod: ubuntu1404 d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] distarch: x86_64 d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] target_arch: x86_64 d20000| 2015-11-04T16:50:59.582-0500 I CONTROL [initandlisten] options: { net: { port: 20000 }, replication: { oplogSizeMB: 40, replSet: "dbhash" }, setParameter: { enableTestCommands: "1" }, storage: { dbPath: "/data/db/dbhash-0", engine: "wiredTiger", mmapv1: { preallocDataFiles: false, smallFiles: true } } } d20000| 2015-11-04T16:50:59.617-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2015-11-04T16:50:59.756-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20000, reason: errno:111 Connection refused 2015-11-04T16:50:59.957-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20000, reason: errno:111 Connection refused d20000| 2015-11-04T16:51:00.065-0500 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger 2015-11-04T16:51:00.158-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20000, reason: errno:111 Connection refused d20000| 2015-11-04T16:51:00.210-0500 I REPL [initandlisten] Did not find local voted for document at startup; NoMatchingDocument Did not find replica set lastVote document in local.replset.election d20000| 2015-11-04T16:51:00.210-0500 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument Did not find replica set configuration document in local.system.replset d20000| 2015-11-04T16:51:00.211-0500 I FTDC [initandlisten] Starting full-time diagnostic data capture with directory '/data/db/dbhash-0/diagnostic.data' d20000| 2015-11-04T16:51:00.211-0500 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker d20000| 2015-11-04T16:51:00.342-0500 I NETWORK [initandlisten] waiting for connections on port 20000 d20000| 2015-11-04T16:51:00.359-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36488 #1 (1 connection now open) [ connection to CAP-JA-Ubuntu14:20000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 20000, 20001, 20002 ] 20001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 20001, "noprealloc" : "", "smallfiles" : "", "replSet" : "dbhash", "dbpath" : "$set-$node", "storageEngine" : "wiredTiger", "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "dbhash" } } ReplSetTest Starting.... Resetting db path '/data/db/dbhash-1' 2015-11-04T16:51:00.362-0500 I - [thread1] shell: started program (sh18649): mongod --oplogSize 40 --port 20001 --noprealloc --smallfiles --replSet dbhash --dbpath /data/db/dbhash-1 --storageEngine wiredTiger --setParameter enableTestCommands=1 2015-11-04T16:51:00.363-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20001, reason: errno:111 Connection refused d20001| note: noprealloc may hurt performance in many applications d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] MongoDB starting : pid=18649 port=20001 dbpath=/data/db/dbhash-1 64-bit host=CAP-JA-Ubuntu14 d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] db version v3.2.0-rc2 d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] git version: 8a3acb42742182c5e314636041c2df368232bbc5 d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014 d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] allocator: tcmalloc d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] modules: none d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] build environment: d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] distmod: ubuntu1404 d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] distarch: x86_64 d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] target_arch: x86_64 d20001| 2015-11-04T16:51:00.392-0500 I CONTROL [initandlisten] options: { net: { port: 20001 }, replication: { oplogSizeMB: 40, replSet: "dbhash" }, setParameter: { enableTestCommands: "1" }, storage: { dbPath: "/data/db/dbhash-1", engine: "wiredTiger", mmapv1: { preallocDataFiles: false, smallFiles: true } } } d20001| 2015-11-04T16:51:00.428-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2015-11-04T16:51:00.564-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20001, reason: errno:111 Connection refused 2015-11-04T16:51:00.765-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20001, reason: errno:111 Connection refused d20001| 2015-11-04T16:51:00.851-0500 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger 2015-11-04T16:51:00.966-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20001, reason: errno:111 Connection refused d20001| 2015-11-04T16:51:01.137-0500 I REPL [initandlisten] Did not find local voted for document at startup; NoMatchingDocument Did not find replica set lastVote document in local.replset.election d20001| 2015-11-04T16:51:01.137-0500 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument Did not find replica set configuration document in local.system.replset d20001| 2015-11-04T16:51:01.137-0500 I FTDC [initandlisten] Starting full-time diagnostic data capture with directory '/data/db/dbhash-1/diagnostic.data' d20001| 2015-11-04T16:51:01.137-0500 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2015-11-04T16:51:01.167-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20001, reason: errno:111 Connection refused d20001| 2015-11-04T16:51:01.262-0500 I NETWORK [initandlisten] waiting for connections on port 20001 d20001| 2015-11-04T16:51:01.367-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:34129 #1 (1 connection now open) [ connection to CAP-JA-Ubuntu14:20000, connection to CAP-JA-Ubuntu14:20001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 20000, 20001, 20002 ] 20002 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 20002, "noprealloc" : "", "smallfiles" : "", "replSet" : "dbhash", "dbpath" : "$set-$node", "storageEngine" : "wiredTiger", "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "dbhash" } } ReplSetTest Starting.... Resetting db path '/data/db/dbhash-2' 2015-11-04T16:51:01.371-0500 I - [thread1] shell: started program (sh18678): mongod --oplogSize 40 --port 20002 --noprealloc --smallfiles --replSet dbhash --dbpath /data/db/dbhash-2 --storageEngine wiredTiger --setParameter enableTestCommands=1 2015-11-04T16:51:01.372-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20002, reason: errno:111 Connection refused d20002| note: noprealloc may hurt performance in many applications d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] MongoDB starting : pid=18678 port=20002 dbpath=/data/db/dbhash-2 64-bit host=CAP-JA-Ubuntu14 d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] db version v3.2.0-rc2 d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] git version: 8a3acb42742182c5e314636041c2df368232bbc5 d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014 d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] allocator: tcmalloc d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] modules: none d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] build environment: d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] distmod: ubuntu1404 d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] distarch: x86_64 d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] target_arch: x86_64 d20002| 2015-11-04T16:51:01.399-0500 I CONTROL [initandlisten] options: { net: { port: 20002 }, replication: { oplogSizeMB: 40, replSet: "dbhash" }, setParameter: { enableTestCommands: "1" }, storage: { dbPath: "/data/db/dbhash-2", engine: "wiredTiger", mmapv1: { preallocDataFiles: false, smallFiles: true } } } d20002| 2015-11-04T16:51:01.434-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2015-11-04T16:51:01.573-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20002, reason: errno:111 Connection refused 2015-11-04T16:51:01.774-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20002, reason: errno:111 Connection refused d20002| 2015-11-04T16:51:01.898-0500 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger 2015-11-04T16:51:01.974-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20002, reason: errno:111 Connection refused d20002| 2015-11-04T16:51:02.093-0500 I REPL [initandlisten] Did not find local voted for document at startup; NoMatchingDocument Did not find replica set lastVote document in local.replset.election d20002| 2015-11-04T16:51:02.093-0500 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument Did not find replica set configuration document in local.system.replset d20002| 2015-11-04T16:51:02.094-0500 I FTDC [initandlisten] Starting full-time diagnostic data capture with directory '/data/db/dbhash-2/diagnostic.data' d20002| 2015-11-04T16:51:02.094-0500 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2015-11-04T16:51:02.175-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:20002, reason: errno:111 Connection refused d20002| 2015-11-04T16:51:02.275-0500 I NETWORK [initandlisten] waiting for connections on port 20002 d20002| 2015-11-04T16:51:02.376-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:46676 #1 (1 connection now open) [ connection to CAP-JA-Ubuntu14:20000, connection to CAP-JA-Ubuntu14:20001, connection to CAP-JA-Ubuntu14:20002 ] { "replSetInitiate" : { "_id" : "dbhash", "members" : [ { "_id" : 0, "host" : "CAP-JA-Ubuntu14:20000" }, { "_id" : 1, "host" : "CAP-JA-Ubuntu14:20001" }, { "_id" : 2, "host" : "CAP-JA-Ubuntu14:20002" } ] } } d20000| 2015-11-04T16:51:02.378-0500 I REPL [conn1] replSetInitiate admin command received from client d20000| 2015-11-04T16:51:02.378-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40760 #2 (2 connections now open) d20001| 2015-11-04T16:51:02.380-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:37570 #2 (2 connections now open) d20000| 2015-11-04T16:51:02.380-0500 I NETWORK [conn2] end connection 127.0.0.1:40760 (1 connection now open) d20002| 2015-11-04T16:51:02.380-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:49343 #2 (2 connections now open) d20001| 2015-11-04T16:51:02.380-0500 I NETWORK [conn2] end connection 127.0.0.1:37570 (1 connection now open) d20000| 2015-11-04T16:51:02.381-0500 I REPL [conn1] replSetInitiate config object with 3 members parses ok d20002| 2015-11-04T16:51:02.381-0500 I NETWORK [conn2] end connection 127.0.0.1:49343 (1 connection now open) d20001| 2015-11-04T16:51:02.382-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:37572 #3 (2 connections now open) d20002| 2015-11-04T16:51:02.382-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:49345 #3 (2 connections now open) d20000| 2015-11-04T16:51:02.383-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20001 d20000| 2015-11-04T16:51:02.383-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20002 d20000| 2015-11-04T16:51:02.383-0500 I REPL [conn1] ****** d20000| 2015-11-04T16:51:02.383-0500 I REPL [conn1] creating replication oplog of size: 40MB... d20000| 2015-11-04T16:51:02.384-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40765 #3 (2 connections now open) d20000| 2015-11-04T16:51:02.384-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40766 #4 (3 connections now open) d20001| 2015-11-04T16:51:02.385-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20000 d20002| 2015-11-04T16:51:02.385-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:02.452-0500 I STORAGE [conn1] Starting WiredTigerRecordStoreThread local.oplog.rs d20000| 2015-11-04T16:51:02.452-0500 I STORAGE [conn1] Scanning the oplog to determine where to place markers for when to truncate d20000| 2015-11-04T16:51:02.843-0500 I REPL [conn1] ****** d20000| 2015-11-04T16:51:03.011-0500 I REPL [ReplicationExecutor] New replica set config in use: { _id: "dbhash", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "CAP-JA-Ubuntu14:20000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "CAP-JA-Ubuntu14:20001", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "CAP-JA-Ubuntu14:20002", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 5000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } d20000| 2015-11-04T16:51:03.011-0500 I REPL [ReplicationExecutor] This node is CAP-JA-Ubuntu14:20000 in the config d20000| 2015-11-04T16:51:03.011-0500 I REPL [ReplicationExecutor] transition to STARTUP2 d20000| 2015-11-04T16:51:03.011-0500 I REPL [conn1] Starting replication applier threads d20000| 2015-11-04T16:51:03.012-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20002 is now in state STARTUP d20000| 2015-11-04T16:51:03.012-0500 I COMMAND [conn1] command local.oplog.rs command: replSetInitiate { replSetInitiate: { _id: "dbhash", members: [ { _id: 0.0, host: "CAP-JA-Ubuntu14:20000" }, { _id: 1.0, host: "CAP-JA-Ubuntu14:20001" }, { _id: 2.0, host: "CAP-JA-Ubuntu14:20002" } ] } } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:22 locks:{ Global: { acquireCount: { r: 5, w: 3, W: 2 }, acquireWaitCount: { W: 1 }, timeAcquiringMicros: { W: 97 } }, Database: { acquireCount: { w: 2, W: 1 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 2 } } } protocol:op_command 634ms d20000| 2015-11-04T16:51:03.014-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20001 is now in state STARTUP d20000| 2015-11-04T16:51:03.016-0500 I REPL [ReplicationExecutor] transition to RECOVERING d20000| 2015-11-04T16:51:03.018-0500 I REPL [ReplicationExecutor] transition to SECONDARY d20000| 2015-11-04T16:51:04.387-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40767 #5 (4 connections now open) d20000| 2015-11-04T16:51:04.387-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40768 #6 (5 connections now open) d20000| 2015-11-04T16:51:04.388-0500 I NETWORK [conn6] end connection 127.0.0.1:40768 (4 connections now open) d20000| 2015-11-04T16:51:04.388-0500 I NETWORK [conn5] end connection 127.0.0.1:40767 (4 connections now open) d20001| 2015-11-04T16:51:04.388-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:37578 #4 (3 connections now open) d20001| 2015-11-04T16:51:04.388-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:37579 #5 (4 connections now open) d20001| 2015-11-04T16:51:04.389-0500 I NETWORK [conn4] end connection 127.0.0.1:37578 (3 connections now open) d20002| 2015-11-04T16:51:04.389-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:49352 #4 (3 connections now open) d20001| 2015-11-04T16:51:04.390-0500 I NETWORK [conn5] end connection 127.0.0.1:37579 (2 connections now open) d20002| 2015-11-04T16:51:04.390-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:49353 #5 (4 connections now open) d20002| 2015-11-04T16:51:04.390-0500 I NETWORK [conn5] end connection 127.0.0.1:49353 (3 connections now open) d20002| 2015-11-04T16:51:04.391-0500 I NETWORK [conn4] end connection 127.0.0.1:49352 (2 connections now open) d20001| 2015-11-04T16:51:04.646-0500 I REPL [replExecDBWorker-0] Starting replication applier threads d20002| 2015-11-04T16:51:04.646-0500 I REPL [replExecDBWorker-0] Starting replication applier threads d20001| 2015-11-04T16:51:04.646-0500 W REPL [rsSync] did not receive a valid config yet d20002| 2015-11-04T16:51:04.646-0500 I REPL [ReplicationExecutor] New replica set config in use: { _id: "dbhash", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "CAP-JA-Ubuntu14:20000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "CAP-JA-Ubuntu14:20001", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "CAP-JA-Ubuntu14:20002", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 5000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } d20002| 2015-11-04T16:51:04.646-0500 I REPL [ReplicationExecutor] This node is CAP-JA-Ubuntu14:20002 in the config d20002| 2015-11-04T16:51:04.646-0500 I REPL [ReplicationExecutor] transition to STARTUP2 d20001| 2015-11-04T16:51:04.647-0500 I REPL [ReplicationExecutor] New replica set config in use: { _id: "dbhash", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "CAP-JA-Ubuntu14:20000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "CAP-JA-Ubuntu14:20001", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "CAP-JA-Ubuntu14:20002", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 5000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } d20001| 2015-11-04T16:51:04.647-0500 I REPL [ReplicationExecutor] This node is CAP-JA-Ubuntu14:20001 in the config d20001| 2015-11-04T16:51:04.647-0500 I REPL [ReplicationExecutor] transition to STARTUP2 d20002| 2015-11-04T16:51:04.647-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20000 is now in state SECONDARY d20001| 2015-11-04T16:51:04.648-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:37583 #6 (3 connections now open) d20001| 2015-11-04T16:51:04.649-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20000 is now in state SECONDARY d20002| 2015-11-04T16:51:04.649-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:49354 #6 (3 connections now open) d20001| 2015-11-04T16:51:04.649-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20002 d20002| 2015-11-04T16:51:04.650-0500 I REPL [rsSync] ****** d20002| 2015-11-04T16:51:04.650-0500 I REPL [rsSync] creating replication oplog of size: 40MB... d20002| 2015-11-04T16:51:04.650-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20001 d20001| 2015-11-04T16:51:04.651-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20002 is now in state STARTUP2 d20002| 2015-11-04T16:51:04.652-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20001 is now in state STARTUP2 d20002| 2015-11-04T16:51:04.777-0500 I STORAGE [rsSync] Starting WiredTigerRecordStoreThread local.oplog.rs d20002| 2015-11-04T16:51:04.777-0500 I STORAGE [rsSync] Scanning the oplog to determine where to place markers for when to truncate d20000| 2015-11-04T16:51:05.017-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20001 is now in state STARTUP2 d20000| 2015-11-04T16:51:05.017-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20002 is now in state STARTUP2 d20002| 2015-11-04T16:51:05.218-0500 I REPL [rsSync] ****** d20002| 2015-11-04T16:51:05.218-0500 I REPL [rsSync] initial sync pending d20002| 2015-11-04T16:51:05.352-0500 I REPL [ReplicationExecutor] syncing from: CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:05.353-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40775 #7 (4 connections now open) d20002| 2015-11-04T16:51:05.359-0500 I REPL [rsSync] initial sync drop all databases d20002| 2015-11-04T16:51:05.359-0500 I STORAGE [rsSync] dropAllDatabasesExceptLocal 1 d20002| 2015-11-04T16:51:05.359-0500 I REPL [rsSync] initial sync clone all databases d20002| 2015-11-04T16:51:05.360-0500 I REPL [rsSync] initial sync data copy, starting syncup d20002| 2015-11-04T16:51:05.360-0500 I REPL [rsSync] oplog sync 1 of 3 d20002| 2015-11-04T16:51:05.360-0500 I REPL [rsSync] oplog sync 2 of 3 d20002| 2015-11-04T16:51:05.360-0500 I REPL [rsSync] initial sync building indexes d20002| 2015-11-04T16:51:05.360-0500 I REPL [rsSync] oplog sync 3 of 3 d20002| 2015-11-04T16:51:05.362-0500 I REPL [rsSync] initial sync finishing up d20002| 2015-11-04T16:51:05.362-0500 I REPL [rsSync] set minValid=(term: -1, timestamp: Nov 4 16:51:03:1) d20002| 2015-11-04T16:51:05.389-0500 I REPL [rsSync] initial sync done d20002| 2015-11-04T16:51:05.392-0500 I REPL [ReplicationExecutor] transition to RECOVERING d20000| 2015-11-04T16:51:05.392-0500 I NETWORK [conn7] end connection 127.0.0.1:40775 (3 connections now open) d20002| 2015-11-04T16:51:05.394-0500 I REPL [ReplicationExecutor] transition to SECONDARY d20001| 2015-11-04T16:51:05.646-0500 I REPL [rsSync] ****** d20001| 2015-11-04T16:51:05.646-0500 I REPL [rsSync] creating replication oplog of size: 40MB... d20002| 2015-11-04T16:51:05.651-0500 I REPL [ReplicationExecutor] could not find member to sync from d20001| 2015-11-04T16:51:05.720-0500 I STORAGE [rsSync] Starting WiredTigerRecordStoreThread local.oplog.rs d20001| 2015-11-04T16:51:05.721-0500 I STORAGE [rsSync] Scanning the oplog to determine where to place markers for when to truncate d20001| 2015-11-04T16:51:06.050-0500 I REPL [rsSync] ****** d20001| 2015-11-04T16:51:06.050-0500 I REPL [rsSync] initial sync pending d20001| 2015-11-04T16:51:06.146-0500 I REPL [ReplicationExecutor] syncing from: CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:06.147-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40776 #8 (4 connections now open) d20001| 2015-11-04T16:51:06.153-0500 I REPL [rsSync] initial sync drop all databases d20001| 2015-11-04T16:51:06.153-0500 I STORAGE [rsSync] dropAllDatabasesExceptLocal 1 d20001| 2015-11-04T16:51:06.153-0500 I REPL [rsSync] initial sync clone all databases d20001| 2015-11-04T16:51:06.154-0500 I REPL [rsSync] initial sync data copy, starting syncup d20001| 2015-11-04T16:51:06.154-0500 I REPL [rsSync] oplog sync 1 of 3 d20001| 2015-11-04T16:51:06.154-0500 I REPL [rsSync] oplog sync 2 of 3 d20001| 2015-11-04T16:51:06.154-0500 I REPL [rsSync] initial sync building indexes d20001| 2015-11-04T16:51:06.154-0500 I REPL [rsSync] oplog sync 3 of 3 d20001| 2015-11-04T16:51:06.156-0500 I REPL [rsSync] initial sync finishing up d20001| 2015-11-04T16:51:06.156-0500 I REPL [rsSync] set minValid=(term: -1, timestamp: Nov 4 16:51:03:1) d20001| 2015-11-04T16:51:06.209-0500 I REPL [rsSync] initial sync done d20000| 2015-11-04T16:51:06.212-0500 I NETWORK [conn8] end connection 127.0.0.1:40776 (3 connections now open) d20001| 2015-11-04T16:51:06.212-0500 I REPL [ReplicationExecutor] transition to RECOVERING d20001| 2015-11-04T16:51:06.214-0500 I REPL [ReplicationExecutor] transition to SECONDARY d20001| 2015-11-04T16:51:06.651-0500 I REPL [ReplicationExecutor] could not find member to sync from d20001| 2015-11-04T16:51:06.652-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20002 is now in state SECONDARY d20000| 2015-11-04T16:51:07.017-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20002 is now in state SECONDARY d20000| 2015-11-04T16:51:07.018-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20001 is now in state SECONDARY d20002| 2015-11-04T16:51:07.652-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20001 is now in state SECONDARY d20000| 2015-11-04T16:51:08.059-0500 I REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected d20001| 2015-11-04T16:51:08.219-0500 I COMMAND [conn3] command local.replset.election command: replSetRequestVotes { replSetRequestVotes: 1, setName: "dbhash", dryRun: true, term: 0, candidateIndex: 0, configVersion: 1, lastCommittedOp: { ts: Timestamp 1446673863000|1, t: -1 } } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:63 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { W: 2 } } } protocol:op_command 159ms d20000| 2015-11-04T16:51:08.219-0500 I REPL [ReplicationExecutor] dry election run succeeded, running for election d20002| 2015-11-04T16:51:08.237-0500 I COMMAND [conn3] command local.replset.election command: replSetRequestVotes { replSetRequestVotes: 1, setName: "dbhash", dryRun: true, term: 0, candidateIndex: 0, configVersion: 1, lastCommittedOp: { ts: Timestamp 1446673863000|1, t: -1 } } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:63 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { W: 2 } } } protocol:op_command 176ms d20002| 2015-11-04T16:51:08.237-0500 I NETWORK [conn3] end connection 127.0.0.1:49345 (2 connections now open) d20000| 2015-11-04T16:51:08.438-0500 I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 1 d20000| 2015-11-04T16:51:08.438-0500 I REPL [ReplicationExecutor] transition to PRIMARY d20002| 2015-11-04T16:51:08.439-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:49358 #7 (3 connections now open) d20002| 2015-11-04T16:51:08.439-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:49359 #8 (4 connections now open) d20000| 2015-11-04T16:51:08.439-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20002 d20000| 2015-11-04T16:51:08.440-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20002 d20001| 2015-11-04T16:51:08.653-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20000 is now in state PRIMARY d20000| 2015-11-04T16:51:09.018-0500 I REPL [rsSync] transition to primary complete; database writes are now permitted remove CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 0 remove CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 6 d20000| 2015-11-04T16:51:09.202-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40779 #9 (4 connections now open) d20000| 2015-11-04T16:51:09.204-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40780 #10 (5 connections now open) remove CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 2 d20000| 2015-11-04T16:51:09.210-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40781 #11 (6 connections now open) remove CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 8 bulkInsert CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 3 d20000| 2015-11-04T16:51:09.218-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40782 #12 (7 connections now open) d20000| 2015-11-04T16:51:09.219-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40783 #13 (8 connections now open) bulkInsert CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 9 remove CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 4 d20000| 2015-11-04T16:51:09.230-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40784 #14 (9 connections now open) bulkInsert CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 5 bulkInsert CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 7 d20000| 2015-11-04T16:51:09.233-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40785 #15 (10 connections now open) d20000| 2015-11-04T16:51:09.235-0500 I NETWORK [conn9] end connection 127.0.0.1:40779 (9 connections now open) d20000| 2015-11-04T16:51:09.241-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40786 #16 (10 connections now open) d20000| 2015-11-04T16:51:09.243-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40787 #17 (11 connections now open) bulkInsert CAP-JA-Ubuntu14:20000 dbhash dbhash_coll 50 1 d20000| 2015-11-04T16:51:09.247-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40788 #18 (12 connections now open) d20000| 2015-11-04T16:51:09.418-0500 I COMMAND [conn14] command dbhash.dbhash_coll command: insert { insert: "dbhash_coll", documents: 1000, ordered: false } ntoreturn:1 ntoskip:0 ninserted:1000 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 1016, w: 1016 } }, Database: { acquireCount: { w: 1016 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 129881 } }, Collection: { acquireCount: { w: 16 } }, Metadata: { acquireCount: { w: 1000 }, acquireWaitCount: { w: 21 }, timeAcquiringMicros: { w: 11090 } }, oplog: { acquireCount: { w: 1000 } } } protocol:op_command 175ms d20000| 2015-11-04T16:51:09.428-0500 I COMMAND [conn13] command dbhash.dbhash_coll command: insert { insert: "dbhash_coll", documents: 1000, ordered: false } ntoreturn:1 ntoskip:0 ninserted:1000 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 1018, w: 1018 } }, Database: { acquireCount: { w: 1017, W: 1 } }, Collection: { acquireCount: { w: 15, W: 1 } }, Metadata: { acquireCount: { w: 1001 }, acquireWaitCount: { w: 22 }, timeAcquiringMicros: { w: 14925 } }, oplog: { acquireCount: { w: 1001 } } } protocol:op_command 187ms d20000| 2015-11-04T16:51:09.435-0500 I COMMAND [conn17] command dbhash.dbhash_coll command: insert { insert: "dbhash_coll", documents: 1000, ordered: false } ntoreturn:1 ntoskip:0 ninserted:1000 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 1016, w: 1016 } }, Database: { acquireCount: { w: 1016 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 118918 } }, Collection: { acquireCount: { w: 16 } }, Metadata: { acquireCount: { w: 1000 }, acquireWaitCount: { w: 22 }, timeAcquiringMicros: { w: 9632 } }, oplog: { acquireCount: { w: 1000 } } } protocol:op_command 181ms d20000| 2015-11-04T16:51:09.449-0500 I COMMAND [conn16] command dbhash.dbhash_coll command: insert { insert: "dbhash_coll", documents: 1000, ordered: false } ntoreturn:1 ntoskip:0 ninserted:1000 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 1016, w: 1016 } }, Database: { acquireCount: { w: 1016 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 120507 } }, Collection: { acquireCount: { w: 16 } }, Metadata: { acquireCount: { w: 1000 }, acquireWaitCount: { w: 25 }, timeAcquiringMicros: { w: 14562 } }, oplog: { acquireCount: { w: 1000 } } } protocol:op_command 182ms d20000| 2015-11-04T16:51:09.461-0500 I COMMAND [conn18] command dbhash.dbhash_coll command: insert { insert: "dbhash_coll", documents: 1000, ordered: false } ntoreturn:1 ntoskip:0 ninserted:1000 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 1016, w: 1016 } }, Database: { acquireCount: { w: 1016 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 115115 } }, Collection: { acquireCount: { w: 16 } }, Metadata: { acquireCount: { w: 1000 }, acquireWaitCount: { w: 21 }, timeAcquiringMicros: { w: 17231 } }, oplog: { acquireCount: { w: 1000 } } } protocol:op_command 204ms d20002| 2015-11-04T16:51:09.652-0500 I REPL [ReplicationExecutor] Member CAP-JA-Ubuntu14:20000 is now in state PRIMARY d20002| 2015-11-04T16:51:10.652-0500 I REPL [ReplicationExecutor] syncing from: CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:10.653-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40789 #19 (13 connections now open) d20002| 2015-11-04T16:51:10.655-0500 I REPL [SyncSourceFeedback] setting syncSourceFeedback to CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:10.665-0500 I NETWORK [conn19] end connection 127.0.0.1:40789 (12 connections now open) d20000| 2015-11-04T16:51:10.669-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40790 #20 (13 connections now open) d20000| 2015-11-04T16:51:10.669-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40791 #21 (14 connections now open) d20002| 2015-11-04T16:51:10.685-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:10.925-0500 I NETWORK [conn14] end connection 127.0.0.1:40784 (13 connections now open) d20000| 2015-11-04T16:51:11.174-0500 I NETWORK [conn13] end connection 127.0.0.1:40783 (12 connections now open) d20000| 2015-11-04T16:51:11.286-0500 I NETWORK [conn16] end connection 127.0.0.1:40786 (11 connections now open) d20000| 2015-11-04T16:51:11.412-0500 I NETWORK [conn18] end connection 127.0.0.1:40788 (10 connections now open) d20000| 2015-11-04T16:51:11.512-0500 I NETWORK [conn17] end connection 127.0.0.1:40787 (9 connections now open) d20001| 2015-11-04T16:51:11.654-0500 I REPL [ReplicationExecutor] syncing from: CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:11.656-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40793 #22 (10 connections now open) d20001| 2015-11-04T16:51:11.680-0500 I REPL [SyncSourceFeedback] setting syncSourceFeedback to CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:11.683-0500 I NETWORK [conn22] end connection 127.0.0.1:40793 (9 connections now open) d20000| 2015-11-04T16:51:11.686-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40794 #23 (10 connections now open) d20000| 2015-11-04T16:51:11.687-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:40795 #24 (11 connections now open) d20001| 2015-11-04T16:51:11.700-0500 I ASIO [NetworkInterfaceASIO] Successfully connected to CAP-JA-Ubuntu14:20000 d20000| 2015-11-04T16:51:35.267-0500 I WRITE [conn11] remove dbhash.dbhash_coll ndeleted:77162 keyUpdates:0 writeConflicts:100052 numYields:100099 locks:{ Global: { acquireCount: { r: 177262, w: 177262 } }, Database: { acquireCount: { w: 177262 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 131058 } }, Collection: { acquireCount: { w: 100100 } }, Metadata: { acquireCount: { w: 77162 }, acquireWaitCount: { w: 98 }, timeAcquiringMicros: { w: 64525 } }, oplog: { acquireCount: { w: 77162 } } } 26025ms d20000| 2015-11-04T16:51:35.267-0500 I WRITE [conn15] remove dbhash.dbhash_coll ndeleted:54287 keyUpdates:0 writeConflicts:105027 numYields:105068 locks:{ Global: { acquireCount: { r: 159356, w: 159356 } }, Database: { acquireCount: { w: 159356 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 130745 } }, Collection: { acquireCount: { w: 105069 } }, Metadata: { acquireCount: { w: 54287 }, acquireWaitCount: { w: 97 }, timeAcquiringMicros: { w: 74202 } }, oplog: { acquireCount: { w: 54287 } } } 26025ms d20000| 2015-11-04T16:51:35.267-0500 I COMMAND [conn11] command dbhash.$cmd command: delete { delete: "dbhash_coll", deletes: [ { q: {}, limit: 0.0 } ], ordered: true } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 177262, w: 177262 } }, Database: { acquireCount: { w: 177262 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 131058 } }, Collection: { acquireCount: { w: 100100 } }, Metadata: { acquireCount: { w: 77162 }, acquireWaitCount: { w: 98 }, timeAcquiringMicros: { w: 64525 } }, oplog: { acquireCount: { w: 77162 } } } protocol:op_command 26025ms d20000| 2015-11-04T16:51:35.267-0500 I COMMAND [conn15] command dbhash.$cmd command: delete { delete: "dbhash_coll", deletes: [ { q: {}, limit: 0.0 } ], ordered: true } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 159356, w: 159356 } }, Database: { acquireCount: { w: 159356 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 130745 } }, Collection: { acquireCount: { w: 105069 } }, Metadata: { acquireCount: { w: 54287 }, acquireWaitCount: { w: 97 }, timeAcquiringMicros: { w: 74202 } }, oplog: { acquireCount: { w: 54287 } } } protocol:op_command 26026ms d20000| 2015-11-04T16:51:35.267-0500 I WRITE [conn10] remove dbhash.dbhash_coll ndeleted:63856 keyUpdates:0 writeConflicts:95657 numYields:95713 locks:{ Global: { acquireCount: { r: 159570, w: 159570 } }, Database: { acquireCount: { w: 159570 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 130624 } }, Collection: { acquireCount: { w: 95714 } }, Metadata: { acquireCount: { w: 63856 }, acquireWaitCount: { w: 86 }, timeAcquiringMicros: { w: 75347 } }, oplog: { acquireCount: { w: 63856 } } } 26026ms d20000| 2015-11-04T16:51:35.267-0500 I COMMAND [conn10] command dbhash.$cmd command: delete { delete: "dbhash_coll", deletes: [ { q: {}, limit: 0.0 } ], ordered: true } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 159570, w: 159570 } }, Database: { acquireCount: { w: 159570 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 130624 } }, Collection: { acquireCount: { w: 95714 } }, Metadata: { acquireCount: { w: 63856 }, acquireWaitCount: { w: 86 }, timeAcquiringMicros: { w: 75347 } }, oplog: { acquireCount: { w: 63856 } } } protocol:op_command 26026ms d20000| 2015-11-04T16:51:35.269-0500 I WRITE [conn12] remove dbhash.dbhash_coll ndeleted:54692 keyUpdates:0 writeConflicts:107513 numYields:107544 locks:{ Global: { acquireCount: { r: 162237, w: 162237 } }, Database: { acquireCount: { w: 162237 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 130396 } }, Collection: { acquireCount: { w: 107545 } }, Metadata: { acquireCount: { w: 54692 }, acquireWaitCount: { w: 78 }, timeAcquiringMicros: { w: 94955 } }, oplog: { acquireCount: { w: 54692 } } } 26027ms d20000| 2015-11-04T16:51:35.270-0500 I COMMAND [conn12] command dbhash.$cmd command: delete { delete: "dbhash_coll", deletes: [ { q: {}, limit: 0.0 } ], ordered: true } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 162237, w: 162237 } }, Database: { acquireCount: { w: 162237 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 130396 } }, Collection: { acquireCount: { w: 107545 } }, Metadata: { acquireCount: { w: 54692 }, acquireWaitCount: { w: 78 }, timeAcquiringMicros: { w: 94955 } }, oplog: { acquireCount: { w: 54692 } } } protocol:op_command 26028ms d20000| 2015-11-04T16:51:35.272-0500 I NETWORK [conn10] end connection 127.0.0.1:40780 (10 connections now open) Waiting for thread 0 completed Waiting for thread 1 completed d20000| 2015-11-04T16:51:35.279-0500 I NETWORK [conn11] end connection 127.0.0.1:40781 (9 connections now open) Waiting for thread 2 completed Waiting for thread 3 completed Waiting for thread 4 completed Waiting for thread 5 completed Waiting for thread 6 completed Waiting for thread 7 completed d20000| 2015-11-04T16:51:35.287-0500 I NETWORK [conn15] end connection 127.0.0.1:40785 (8 connections now open) d20000| 2015-11-04T16:51:35.288-0500 I NETWORK [conn12] end connection 127.0.0.1:40782 (7 connections now open) Waiting for thread 8 completed Waiting for thread 9 completed ---- CAP-JA-Ubuntu14:20000: awaitReplication started ---- d20000| 2015-11-04T16:51:35.717-0500 I COMMAND [conn1] command test.teardown command: insert { insert: "teardown", documents: [ { _id: ObjectId('563a7de7812b8fb77245e80d'), a: 1.0 } ], ordered: true, writeConcern: { w: 3.0, wtimeout: 300000.0 } } ntoreturn:1 ntoskip:0 ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:85 locks:{ Global: { acquireCount: { r: 4, w: 4 } }, Database: { acquireCount: { w: 3, W: 1 } }, Collection: { acquireCount: { W: 1 } }, Metadata: { acquireCount: { w: 2 } }, oplog: { acquireCount: { w: 2 } } } protocol:op_command 427ms ---- CAP-JA-Ubuntu14:20000: awaitReplication completed in 429 ms ---- dbHash for db dbhash { "master" : { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20000", "collections" : { "dbhash_coll" : "d41d8cd98f00b204e9800998ecf8427e" }, "md5" : "74be16979710d4c4e7c6647856088456", "timeMillis" : 72, "fromCache" : [ ], "ok" : 1 }, "slaves" : [ { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20001", "collections" : { "dbhash_coll" : "d41d8cd98f00b204e9800998ecf8427e" }, "md5" : "74be16979710d4c4e7c6647856088456", "timeMillis" : 78, "fromCache" : [ ], "ok" : 1 }, { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20002", "collections" : { "dbhash_coll" : "cc377c5970ac33b6c2246ca4e770cb21" }, "md5" : "ea7468301789e03252dedc0b20bc3d4c", "timeMillis" : 59, "fromCache" : [ ], "ok" : 1 } ] } primary CAP-JA-Ubuntu14:20000 0 primary last oplog entry { "ts" : Timestamp(1446673895, 2788), "t" : NumberLong(1), "h" : NumberLong("-6905531911049596245"), "v" : 2, "op" : "i", "ns" : "test.teardown", "o" : { "_id" : ObjectId("563a7de7812b8fb77245e80d"), "a" : 1 } } secondary CAP-JA-Ubuntu14:20001 0 secondary last oplog entry { "ts" : Timestamp(1446673895, 2788), "t" : NumberLong(1), "h" : NumberLong("-6905531911049596245"), "v" : 2, "op" : "i", "ns" : "test.teardown", "o" : { "_id" : ObjectId("563a7de7812b8fb77245e80d"), "a" : 1 } } secondary CAP-JA-Ubuntu14:20002 1 secondary last oplog entry { "ts" : Timestamp(1446673895, 2788), "t" : NumberLong(1), "h" : NumberLong("-6905531911049596245"), "v" : 2, "op" : "i", "ns" : "test.teardown", "o" : { "_id" : ObjectId("563a7de7812b8fb77245e80d"), "a" : 1 } } assert: ["74be16979710d4c4e7c6647856088456"] != ["ea7468301789e03252dedc0b20bc3d4c"] are not equal : dbHash inconsistency for db dbhash { "master" : { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20000", "collections" : { "dbhash_coll" : "d41d8cd98f00b204e9800998ecf8427e" }, "md5" : "74be16979710d4c4e7c6647856088456", "timeMillis" : 72, "fromCache" : [ ], "ok" : 1 }, "slaves" : [ { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20001", "collections" : { "dbhash_coll" : "d41d8cd98f00b204e9800998ecf8427e" }, "md5" : "74be16979710d4c4e7c6647856088456", "timeMillis" : 78, "fromCache" : [ ], "ok" : 1 }, { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20002", "collections" : { "dbhash_coll" : "cc377c5970ac33b6c2246ca4e770cb21" }, "md5" : "ea7468301789e03252dedc0b20bc3d4c", "timeMillis" : 59, "fromCache" : [ ], "ok" : 1 } ] } doassert@src/mongo/shell/assert.js:15:14 assert.eq@src/mongo/shell/assert.js:49:5 @../dbhash-remove_and_bulk_insert.js:110:5 @../dbhash-remove_and_bulk_insert.js:97:1 2015-11-04T16:51:36.085-0500 E QUERY [thread1] Error: ["74be16979710d4c4e7c6647856088456"] != ["ea7468301789e03252dedc0b20bc3d4c"] are not equal : dbHash inconsistency for db dbhash { "master" : { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20000", "collections" : { "dbhash_coll" : "d41d8cd98f00b204e9800998ecf8427e" }, "md5" : "74be16979710d4c4e7c6647856088456", "timeMillis" : 72, "fromCache" : [ ], "ok" : 1 }, "slaves" : [ { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20001", "collections" : { "dbhash_coll" : "d41d8cd98f00b204e9800998ecf8427e" }, "md5" : "74be16979710d4c4e7c6647856088456", "timeMillis" : 78, "fromCache" : [ ], "ok" : 1 }, { "numCollections" : 1, "host" : "CAP-JA-Ubuntu14:20002", "collections" : { "dbhash_coll" : "cc377c5970ac33b6c2246ca4e770cb21" }, "md5" : "ea7468301789e03252dedc0b20bc3d4c", "timeMillis" : 59, "fromCache" : [ ], "ok" : 1 } ] } : doassert@src/mongo/shell/assert.js:15:14 assert.eq@src/mongo/shell/assert.js:49:5 @../dbhash-remove_and_bulk_insert.js:110:5 @../dbhash-remove_and_bulk_insert.js:97:1 failed to load: ../dbhash-remove_and_bulk_insert.js d20002| 2015-11-04T16:51:36.085-0500 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends d20002| 2015-11-04T16:51:36.085-0500 I FTDC [signalProcessingThread] Stopping full-time diagnostic data capture d20002| 2015-11-04T16:51:36.088-0500 I REPL [signalProcessingThread] Stopping replication applier threads d20002| 2015-11-04T16:51:36.449-0500 I STORAGE [conn8] got request after shutdown() d20000| 2015-11-04T16:51:36.450-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable End of file d20002| 2015-11-04T16:51:36.450-0500 I STORAGE [conn7] got request after shutdown() d20000| 2015-11-04T16:51:36.450-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable End of file d20002| 2015-11-04T16:51:37.157-0500 I STORAGE [conn6] got request after shutdown() d20001| 2015-11-04T16:51:37.157-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable End of file d20002| 2015-11-04T16:51:38.218-0500 W EXECUTOR [rsBackgroundSync] killCursors command task failed: CallbackCanceled Callback canceled d20000| 2015-11-04T16:51:38.220-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20002 - HostUnreachable Connection reset by peer d20001| 2015-11-04T16:51:38.220-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20002 - HostUnreachable Connection reset by peer d20002| 2015-11-04T16:51:38.220-0500 I CONTROL [signalProcessingThread] now exiting d20002| 2015-11-04T16:51:38.220-0500 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... d20002| 2015-11-04T16:51:38.220-0500 I NETWORK [signalProcessingThread] closing listening socket: 14 d20002| 2015-11-04T16:51:38.220-0500 I NETWORK [signalProcessingThread] closing listening socket: 15 d20002| 2015-11-04T16:51:38.220-0500 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20002.sock d20002| 2015-11-04T16:51:38.220-0500 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog... d20002| 2015-11-04T16:51:38.220-0500 I NETWORK [signalProcessingThread] shutdown: going to close sockets... d20002| 2015-11-04T16:51:38.220-0500 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down d20002| 2015-11-04T16:51:38.220-0500 I NETWORK [conn1] end connection 127.0.0.1:46676 (0 connections now open) d20001| 2015-11-04T16:51:38.220-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable Connection reset by peer d20000| 2015-11-04T16:51:38.221-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable Connection reset by peer d20000| 2015-11-04T16:51:38.221-0500 I NETWORK [conn21] end connection 127.0.0.1:40791 (6 connections now open) d20001| 2015-11-04T16:51:38.221-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20002 - HostUnreachable Connection refused d20001| 2015-11-04T16:51:38.221-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable Connection refused d20002| 2015-11-04T16:51:39.073-0500 I STORAGE [signalProcessingThread] shutdown: removing fs lock... d20002| 2015-11-04T16:51:39.074-0500 I CONTROL [signalProcessingThread] dbexit: rc: 0 d20000| 2015-11-04T16:51:39.093-0500 I NETWORK [conn4] end connection 127.0.0.1:40766 (5 connections now open) d20000| 2015-11-04T16:51:39.093-0500 I NETWORK [conn20] end connection 127.0.0.1:40790 (5 connections now open) d20001| 2015-11-04T16:51:39.093-0500 I NETWORK [conn6] end connection 127.0.0.1:37583 (2 connections now open) d20001| 2015-11-04T16:51:40.086-0500 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends d20001| 2015-11-04T16:51:40.086-0500 I FTDC [signalProcessingThread] Stopping full-time diagnostic data capture d20001| 2015-11-04T16:51:40.089-0500 I REPL [signalProcessingThread] Stopping replication applier threads d20000| 2015-11-04T16:51:40.222-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20002 - HostUnreachable Connection refused d20000| 2015-11-04T16:51:40.222-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable Connection refused d20000| 2015-11-04T16:51:40.222-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20002 - HostUnreachable Connection refused d20000| 2015-11-04T16:51:40.222-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable Connection refused d20000| 2015-11-04T16:51:40.223-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20002 - HostUnreachable Connection refused d20000| 2015-11-04T16:51:40.223-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20002; HostUnreachable Connection refused d20001| 2015-11-04T16:51:40.449-0500 I STORAGE [conn3] got request after shutdown() d20000| 2015-11-04T16:51:40.450-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20001; HostUnreachable End of file d20001| 2015-11-04T16:51:40.718-0500 W EXECUTOR [rsBackgroundSync] killCursors command task failed: CallbackCanceled Callback canceled d20000| 2015-11-04T16:51:40.720-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20001 - HostUnreachable Connection reset by peer d20001| 2015-11-04T16:51:40.720-0500 I CONTROL [signalProcessingThread] now exiting d20001| 2015-11-04T16:51:40.720-0500 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... d20001| 2015-11-04T16:51:40.720-0500 I NETWORK [signalProcessingThread] closing listening socket: 11 d20001| 2015-11-04T16:51:40.720-0500 I NETWORK [signalProcessingThread] closing listening socket: 12 d20001| 2015-11-04T16:51:40.720-0500 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20001.sock d20001| 2015-11-04T16:51:40.720-0500 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog... d20001| 2015-11-04T16:51:40.720-0500 I NETWORK [signalProcessingThread] shutdown: going to close sockets... d20001| 2015-11-04T16:51:40.720-0500 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down d20001| 2015-11-04T16:51:40.721-0500 I NETWORK [conn1] end connection 127.0.0.1:34129 (0 connections now open) d20000| 2015-11-04T16:51:40.721-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20001; HostUnreachable Connection reset by peer d20000| 2015-11-04T16:51:40.721-0500 I NETWORK [conn23] end connection 127.0.0.1:40794 (3 connections now open) d20000| 2015-11-04T16:51:40.721-0500 I ASIO [NetworkInterfaceASIO] Failed to connect to CAP-JA-Ubuntu14:20001 - HostUnreachable Connection refused d20000| 2015-11-04T16:51:40.722-0500 I REPL [ReplicationExecutor] Error in heartbeat request to CAP-JA-Ubuntu14:20001; HostUnreachable Connection refused d20001| 2015-11-04T16:51:41.392-0500 I STORAGE [signalProcessingThread] shutdown: removing fs lock... d20001| 2015-11-04T16:51:41.393-0500 I CONTROL [signalProcessingThread] dbexit: rc: 0 d20000| 2015-11-04T16:51:41.415-0500 I NETWORK [conn24] end connection 127.0.0.1:40795 (2 connections now open) d20000| 2015-11-04T16:51:41.415-0500 I NETWORK [conn3] end connection 127.0.0.1:40765 (2 connections now open) d20000| 2015-11-04T16:51:42.086-0500 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends d20000| 2015-11-04T16:51:42.086-0500 I FTDC [signalProcessingThread] Stopping full-time diagnostic data capture d20000| 2015-11-04T16:51:42.089-0500 I REPL [signalProcessingThread] Stopping replication applier threads d20000| 2015-11-04T16:51:43.023-0500 I CONTROL [signalProcessingThread] now exiting d20000| 2015-11-04T16:51:43.023-0500 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... d20000| 2015-11-04T16:51:43.023-0500 I NETWORK [signalProcessingThread] closing listening socket: 8 d20000| 2015-11-04T16:51:43.023-0500 I NETWORK [signalProcessingThread] closing listening socket: 9 d20000| 2015-11-04T16:51:43.023-0500 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20000.sock d20000| 2015-11-04T16:51:43.023-0500 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog... d20000| 2015-11-04T16:51:43.023-0500 I NETWORK [signalProcessingThread] shutdown: going to close sockets... d20000| 2015-11-04T16:51:43.023-0500 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down d20000| 2015-11-04T16:51:43.023-0500 I NETWORK [conn1] end connection 127.0.0.1:36488 (0 connections now open) d20000| 2015-11-04T16:51:43.621-0500 I STORAGE [signalProcessingThread] shutdown: removing fs lock... d20000| 2015-11-04T16:51:43.621-0500 I CONTROL [signalProcessingThread] dbexit: rc: 0