2014-12-23T16:45:23.380+0000 I CONTROL [initandlisten] MongoDB starting : pid=12865 port=27018 dbpath=/mnt/data/mongod-shard 64-bit host=ip-10-238-44-130 2014-12-23T16:45:23.380+0000 I CONTROL [initandlisten] db version v2.8.0-rc3 2014-12-23T16:45:23.380+0000 I CONTROL [initandlisten] git version: 2d679247f17dab05a492c8b6d2c250dab18e54f2 2014-12-23T16:45:23.380+0000 I CONTROL [initandlisten] build info: Linux build4.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49 2014-12-23T16:45:23.380+0000 I CONTROL [initandlisten] allocator: tcmalloc 2014-12-23T16:45:23.380+0000 I CONTROL [initandlisten] options: { net: { port: 27018 }, processManagement: { fork: true }, storage: { dbPath: "/mnt/data/mongod-shard", engine: "wiredtiger" }, systemLog: { destination: "file", logAppend: true, path: "/mnt/data/mongod-shard/mongod-shard.log" } } 2014-12-23T16:45:23.380+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,extensions=[local=(entry=index_collator_extension)],statistics=(all),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2014-12-23T16:45:24.271+0000 I NETWORK [initandlisten] waiting for connections on port 27018 2014-12-23T16:51:32.492+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:41419 #1 (1 connection now open) 2014-12-23T16:52:00.181+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:41426 #2 (2 connections now open) 2014-12-23T16:52:00.186+0000 I INDEX [conn2] build index on: benchdb1.COL-0 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-0" } 2014-12-23T16:52:00.186+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:00.187+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:00.480+0000 I SHARDING [conn2] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers 2014-12-23T16:52:00.483+0000 I SHARDING [conn2] remote client 10.233.133.124:41426 initialized this host (ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018) as shard shard0002 2014-12-23T16:52:00.483+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 1|2||54999dad00df49298418fe62, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:00.483+0000 I NETWORK [conn2] SyncClusterConnection connecting to [ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:00.548+0000 I NETWORK [conn2] SyncClusterConnection connecting to [ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:00.548+0000 I NETWORK [conn2] SyncClusterConnection connecting to [ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:00.550+0000 I SHARDING [conn2] collection benchdb1.COL-0 was previously unsharded, new metadata loaded with shard version 1|2||54999dad00df49298418fe62 2014-12-23T16:52:00.550+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999dad00df49298418fe62, took 67ms 2014-12-23T16:52:00.583+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:00.584+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dad00df49298418fe62') } 2014-12-23T16:52:00.591+0000 I SHARDING [LockPinger] creating distributed lock ping thread for ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 and process ip-10-238-44-130:27018:1419353520:1363056040 (sleeping for 30000ms) 2014-12-23T16:52:00.591+0000 I NETWORK [LockPinger] SyncClusterConnection connecting to [ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:00.592+0000 I NETWORK [LockPinger] SyncClusterConnection connecting to [ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:00.592+0000 I NETWORK [LockPinger] SyncClusterConnection connecting to [ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:01.149+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:52:00 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:52:01.453+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999db09a8f659f34bfb227 2014-12-23T16:52:01.453+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:01-54999db19a8f659f34bfb228", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353521453), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:01.828+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 1|2||54999dad00df49298418fe62, current metadata version is 1|2||54999dad00df49298418fe62 2014-12-23T16:52:01.829+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-0 already up to date (shard version : 1|2||54999dad00df49298418fe62, took 0ms) 2014-12-23T16:52:01.829+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999dad00df49298418fe62 2014-12-23T16:52:01.829+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:02.089+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:02.092+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:02.096+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:02.104+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:02.121+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:02.153+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:02.160+0000 I NETWORK [initandlisten] connection accepted from 10.238.1.75:48430 #3 (3 connections now open) 2014-12-23T16:52:02.218+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:02.218+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:02.218+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:02.218+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999dad00df49298418fe62 2014-12-23T16:52:02.582+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:02.582+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999dad00df49298418fe62 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T16:52:02.582+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:02.583+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:02.583+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:02.891+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:02-54999db29a8f659f34bfb229", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353522891), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:02.947+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:41427 #4 (4 connections now open) 2014-12-23T16:52:02.952+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:02.952+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:02.952+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 30 2014-12-23T16:52:02.952+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:02.952+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:02.952+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:02.953+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:02.954+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019] 2014-12-23T16:52:03.257+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:03.257+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:03-54999db39a8f659f34bfb22a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353523257), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 1245, step 3 of 6: 258, step 4 of 6: 129, step 5 of 6: 734, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:03.345+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dad00df49298418fe62') } ntoreturn:1 keyUpdates:0 reslen:37 2761ms 2014-12-23T16:52:03.347+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:03.347+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dad00df49298418fe62') } 2014-12-23T16:52:03.525+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('54999dad00df49298418fe62') } 2014-12-23T16:52:03.526+0000 W SHARDING [conn4] could not acquire collection lock for benchdb1.COL-0 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-0 is taken. 2014-12-23T16:52:03.526+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:03-54999db39a8f659f34bfb22c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353523526), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-0 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey..." } } 2014-12-23T16:52:03.824+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999db39a8f659f34bfb22b 2014-12-23T16:52:03.824+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:03-54999db39a8f659f34bfb22d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353523824), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:03.879+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 2|0||54999dad00df49298418fe62, current metadata version is 2|0||54999dad00df49298418fe62 2014-12-23T16:52:03.880+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 2|0||54999dad00df49298418fe62 to shard version 2|1||54999dad00df49298418fe62 2014-12-23T16:52:03.880+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999dad00df49298418fe62, took 0ms 2014-12-23T16:52:03.880+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999dad00df49298418fe62 2014-12-23T16:52:03.880+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:04.276+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.279+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.283+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.292+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.308+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.341+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.390+0000 I NETWORK [initandlisten] connection accepted from 10.238.4.248:55181 #5 (5 connections now open) 2014-12-23T16:52:04.405+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.534+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:04.534+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:04.534+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:04.534+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999dad00df49298418fe62 2014-12-23T16:52:04.907+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:04.907+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999dad00df49298418fe62 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T16:52:05.212+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:05-54999db59a8f659f34bfb22e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353525212), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:05.267+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:05.267+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:05.267+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 47 2014-12-23T16:52:05.267+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:05.267+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:05.570+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:05.570+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:05-54999db59a8f659f34bfb22f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353525570), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 533, step 3 of 6: 394, step 4 of 6: 258, step 5 of 6: 732, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:05.627+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dad00df49298418fe62') } ntoreturn:1 keyUpdates:0 reslen:37 2279ms 2014-12-23T16:52:05.629+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dad00df49298418fe62') } 2014-12-23T16:52:06.236+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999db59a8f659f34bfb230 2014-12-23T16:52:06.237+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 3|0||54999dad00df49298418fe62, current metadata version is 3|0||54999dad00df49298418fe62 2014-12-23T16:52:06.237+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 3|0||54999dad00df49298418fe62 to shard version 3|1||54999dad00df49298418fe62 2014-12-23T16:52:06.237+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999dad00df49298418fe62, took 0ms 2014-12-23T16:52:06.237+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999dad00df49298418fe62 2014-12-23T16:52:06.543+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:06-54999db69a8f659f34bfb231", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353526543), what: "split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999dad00df49298418fe62') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999dad00df49298418fe62') } } } 2014-12-23T16:52:06.902+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:06.902+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dad00df49298418fe62') } ntoreturn:1 keyUpdates:0 reslen:37 1273ms 2014-12-23T16:52:09.413+0000 I INDEX [conn2] build index on: benchdb1.COL-1 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-1" } 2014-12-23T16:52:09.413+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:09.414+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:09.710+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 1|2||54999db600df49298418fe67, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:09.711+0000 I SHARDING [conn2] collection benchdb1.COL-1 was previously unsharded, new metadata loaded with shard version 1|2||54999db600df49298418fe67 2014-12-23T16:52:09.711+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999db600df49298418fe67, took 0ms 2014-12-23T16:52:09.745+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:09.745+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999db600df49298418fe67') } 2014-12-23T16:52:10.656+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dba9a8f659f34bfb233 2014-12-23T16:52:10.656+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:10-54999dba9a8f659f34bfb234", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353530656), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:10.719+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||54999db600df49298418fe67, current metadata version is 1|2||54999db600df49298418fe67 2014-12-23T16:52:10.720+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||54999db600df49298418fe67, took 0ms) 2014-12-23T16:52:10.720+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999db600df49298418fe67 2014-12-23T16:52:10.720+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:10.723+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:10.725+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:10.730+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:10.730+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:10.730+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:10.730+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999db600df49298418fe67 2014-12-23T16:52:10.781+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:10.781+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999db600df49298418fe67 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T16:52:11.086+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:11-54999dbb9a8f659f34bfb235", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353531086), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:11.140+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:11.140+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:11.140+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 81 2014-12-23T16:52:11.140+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:11.140+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:11.443+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:11.444+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:11-54999dbb9a8f659f34bfb236", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353531444), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 974, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 409, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:11.491+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999db600df49298418fe67') } ntoreturn:1 keyUpdates:0 reslen:37 1746ms 2014-12-23T16:52:11.493+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:11.493+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999db600df49298418fe67') } 2014-12-23T16:52:12.100+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dbb9a8f659f34bfb237 2014-12-23T16:52:12.101+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:12-54999dbc9a8f659f34bfb238", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353532101), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:12.165+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 2|0||54999db600df49298418fe67, current metadata version is 2|0||54999db600df49298418fe67 2014-12-23T16:52:12.166+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 2|0||54999db600df49298418fe67 to shard version 2|1||54999db600df49298418fe67 2014-12-23T16:52:12.166+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999db600df49298418fe67, took 0ms 2014-12-23T16:52:12.166+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999db600df49298418fe67 2014-12-23T16:52:12.166+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:12.169+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:12.171+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:12.176+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:12.184+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:12.200+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:12.200+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:12.201+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:12.201+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999db600df49298418fe67 2014-12-23T16:52:12.263+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:12.263+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999db600df49298418fe67 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T16:52:12.568+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:12-54999dbc9a8f659f34bfb239", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353532568), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:12.625+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:12.625+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:12.625+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 89 2014-12-23T16:52:12.625+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:12.625+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:12.928+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:12.928+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:12-54999dbc9a8f659f34bfb23a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353532928), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:12.984+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999db600df49298418fe67') } ntoreturn:1 keyUpdates:0 reslen:37 1491ms 2014-12-23T16:52:12.986+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999db600df49298418fe67') } 2014-12-23T16:52:13.593+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dbc9a8f659f34bfb23b 2014-12-23T16:52:13.593+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 3|0||54999db600df49298418fe67, current metadata version is 3|0||54999db600df49298418fe67 2014-12-23T16:52:13.594+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 3|0||54999db600df49298418fe67 to shard version 3|1||54999db600df49298418fe67 2014-12-23T16:52:13.594+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999db600df49298418fe67, took 0ms 2014-12-23T16:52:13.594+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999db600df49298418fe67 2014-12-23T16:52:13.881+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:13-54999dbd9a8f659f34bfb23c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353533881), what: "split", ns: "benchdb1.COL-1", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999db600df49298418fe67') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999db600df49298418fe67') } } } 2014-12-23T16:52:14.215+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:14.215+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999db600df49298418fe67') } ntoreturn:1 keyUpdates:0 reslen:37 1229ms 2014-12-23T16:52:16.639+0000 I INDEX [conn2] build index on: benchdb1.COL-2 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-2" } 2014-12-23T16:52:16.639+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:16.640+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:16.921+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-2 with requested shard version 1|2||54999dbd00df49298418fe6c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:16.921+0000 I SHARDING [conn2] collection benchdb1.COL-2 was previously unsharded, new metadata loaded with shard version 1|2||54999dbd00df49298418fe6c 2014-12-23T16:52:16.921+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999dbd00df49298418fe6c, took 0ms 2014-12-23T16:52:16.959+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:16.960+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dbd00df49298418fe6c') } 2014-12-23T16:52:17.870+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dc19a8f659f34bfb23e 2014-12-23T16:52:17.870+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:17-54999dc19a8f659f34bfb23f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353537870), what: "moveChunk.start", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:17.927+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 1|2||54999dbd00df49298418fe6c, current metadata version is 1|2||54999dbd00df49298418fe6c 2014-12-23T16:52:17.928+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-2 already up to date (shard version : 1|2||54999dbd00df49298418fe6c, took 0ms) 2014-12-23T16:52:17.928+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999dbd00df49298418fe6c 2014-12-23T16:52:17.929+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:17.931+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:17.934+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:17.939+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:17.948+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:17.948+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:17.948+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:17.948+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999dbd00df49298418fe6c 2014-12-23T16:52:17.992+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:17.992+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999dbd00df49298418fe6c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-2' 2014-12-23T16:52:18.298+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:18-54999dc29a8f659f34bfb240", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353538298), what: "moveChunk.commit", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:18.357+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:18.357+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:18.357+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-2 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 110 2014-12-23T16:52:18.357+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-2 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:18.357+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:18.660+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:18.660+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:18-54999dc29a8f659f34bfb241", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353538660), what: "moveChunk.from", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 968, step 3 of 6: 1, step 4 of 6: 17, step 5 of 6: 409, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:18.710+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dbd00df49298418fe6c') } ntoreturn:1 keyUpdates:0 reslen:37 1751ms 2014-12-23T16:52:18.712+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:18.712+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dbd00df49298418fe6c') } 2014-12-23T16:52:19.320+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dc29a8f659f34bfb242 2014-12-23T16:52:19.320+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:19-54999dc39a8f659f34bfb243", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353539320), what: "moveChunk.start", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:19.374+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 2|0||54999dbd00df49298418fe6c, current metadata version is 2|0||54999dbd00df49298418fe6c 2014-12-23T16:52:19.375+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-2 from shard version 2|0||54999dbd00df49298418fe6c to shard version 2|1||54999dbd00df49298418fe6c 2014-12-23T16:52:19.375+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999dbd00df49298418fe6c, took 0ms 2014-12-23T16:52:19.375+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999dbd00df49298418fe6c 2014-12-23T16:52:19.375+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:19.377+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:19.379+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:19.384+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:19.392+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:19.409+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:19.409+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:19.410+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:19.410+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999dbd00df49298418fe6c 2014-12-23T16:52:19.461+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:19.461+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999dbd00df49298418fe6c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-2' 2014-12-23T16:52:19.766+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:19-54999dc39a8f659f34bfb244", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353539766), what: "moveChunk.commit", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:19.817+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:19.817+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:19.817+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-2 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 118 2014-12-23T16:52:19.817+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-2 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:19.817+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:20.074+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:20.074+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:20-54999dc49a8f659f34bfb245", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353540074), what: "moveChunk.from", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 407, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:20.124+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dbd00df49298418fe6c') } ntoreturn:1 keyUpdates:0 reslen:37 1412ms 2014-12-23T16:52:20.126+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-2", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-2-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dbd00df49298418fe6c') } 2014-12-23T16:52:20.696+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dc49a8f659f34bfb246 2014-12-23T16:52:20.696+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 3|0||54999dbd00df49298418fe6c, current metadata version is 3|0||54999dbd00df49298418fe6c 2014-12-23T16:52:20.697+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-2 from shard version 3|0||54999dbd00df49298418fe6c to shard version 3|1||54999dbd00df49298418fe6c 2014-12-23T16:52:20.697+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999dbd00df49298418fe6c, took 0ms 2014-12-23T16:52:20.697+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999dbd00df49298418fe6c 2014-12-23T16:52:21.002+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:20-54999dc49a8f659f34bfb247", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353541001), what: "split", ns: "benchdb1.COL-2", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999dbd00df49298418fe6c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999dbd00df49298418fe6c') } } } 2014-12-23T16:52:21.391+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:21.391+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-2", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-2-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dbd00df49298418fe6c') } ntoreturn:1 keyUpdates:0 reslen:37 1264ms 2014-12-23T16:52:23.997+0000 I INDEX [conn2] build index on: benchdb1.COL-3 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-3" } 2014-12-23T16:52:23.997+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:23.998+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:24.326+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-3 with requested shard version 1|2||54999dc500df49298418fe6f, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:24.326+0000 I SHARDING [conn2] collection benchdb1.COL-3 was previously unsharded, new metadata loaded with shard version 1|2||54999dc500df49298418fe6f 2014-12-23T16:52:24.326+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999dc500df49298418fe6f, took 0ms 2014-12-23T16:52:24.366+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:24.367+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dc500df49298418fe6f') } 2014-12-23T16:52:25.152+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dc89a8f659f34bfb249 2014-12-23T16:52:25.152+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:25-54999dc99a8f659f34bfb24a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353545152), what: "moveChunk.start", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:25.218+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 1|2||54999dc500df49298418fe6f, current metadata version is 1|2||54999dc500df49298418fe6f 2014-12-23T16:52:25.219+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-3 already up to date (shard version : 1|2||54999dc500df49298418fe6f, took 0ms) 2014-12-23T16:52:25.219+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999dc500df49298418fe6f 2014-12-23T16:52:25.219+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:25.222+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:25.224+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:25.229+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:25.229+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:25.229+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:25.229+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999dc500df49298418fe6f 2014-12-23T16:52:25.292+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:25.292+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999dc500df49298418fe6f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-3' 2014-12-23T16:52:25.443+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('54999dc500df49298418fe6f') } 2014-12-23T16:52:25.443+0000 W SHARDING [conn4] migration already in progress 2014-12-23T16:52:25.443+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:25-54999dc99a8f659f34bfb24b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353545443), what: "moveChunk.from", ns: "benchdb1.COL-3", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T16:52:25.468+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:25-54999dc99a8f659f34bfb24c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353545468), what: "moveChunk.commit", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:25.538+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:25.538+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:25.538+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-3 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 137 2014-12-23T16:52:25.539+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-3 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:25.539+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:25.746+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:25.746+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:25-54999dc99a8f659f34bfb24d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353545746), what: "moveChunk.from", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 852, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 309, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:25.804+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dc500df49298418fe6f') } ntoreturn:1 keyUpdates:0 reslen:37 1437ms 2014-12-23T16:52:25.805+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:25.805+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dc500df49298418fe6f') } 2014-12-23T16:52:26.412+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dc99a8f659f34bfb24e 2014-12-23T16:52:26.412+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:26-54999dca9a8f659f34bfb24f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353546412), what: "moveChunk.start", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:26.470+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 2|0||54999dc500df49298418fe6f, current metadata version is 2|0||54999dc500df49298418fe6f 2014-12-23T16:52:26.470+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-3 from shard version 2|0||54999dc500df49298418fe6f to shard version 2|1||54999dc500df49298418fe6f 2014-12-23T16:52:26.470+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999dc500df49298418fe6f, took 0ms 2014-12-23T16:52:26.470+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999dc500df49298418fe6f 2014-12-23T16:52:26.471+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:26.473+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:26.475+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:26.480+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:26.488+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:26.504+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:26.504+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:26.505+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:26.505+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999dc500df49298418fe6f 2014-12-23T16:52:26.548+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:26.548+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999dc500df49298418fe6f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-3' 2014-12-23T16:52:26.853+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:26-54999dca9a8f659f34bfb250", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353546853), what: "moveChunk.commit", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:26.909+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:26.909+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:26.909+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-3 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 148 2014-12-23T16:52:26.909+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-3 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:26.909+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:27.212+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:27.212+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:27-54999dcb9a8f659f34bfb251", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353547212), what: "moveChunk.from", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 665, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 404, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:27.265+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dc500df49298418fe6f') } ntoreturn:1 keyUpdates:0 reslen:37 1459ms 2014-12-23T16:52:27.266+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-3", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-3-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dc500df49298418fe6f') } 2014-12-23T16:52:27.873+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dcb9a8f659f34bfb252 2014-12-23T16:52:27.873+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 3|0||54999dc500df49298418fe6f, current metadata version is 3|0||54999dc500df49298418fe6f 2014-12-23T16:52:27.874+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-3 from shard version 3|0||54999dc500df49298418fe6f to shard version 3|1||54999dc500df49298418fe6f 2014-12-23T16:52:27.874+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999dc500df49298418fe6f, took 0ms 2014-12-23T16:52:27.874+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999dc500df49298418fe6f 2014-12-23T16:52:28.178+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:28-54999dcc9a8f659f34bfb253", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353548178), what: "split", ns: "benchdb1.COL-3", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999dc500df49298418fe6f') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999dc500df49298418fe6f') } } } 2014-12-23T16:52:28.566+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:28.566+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-3", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-3-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dc500df49298418fe6f') } ntoreturn:1 keyUpdates:0 reslen:37 1299ms 2014-12-23T16:52:31.173+0000 I INDEX [conn2] build index on: benchdb1.COL-4 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-4" } 2014-12-23T16:52:31.173+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:31.173+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:31.477+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:52:31 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:52:31.478+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-4 with requested shard version 1|2||54999dcc00df49298418fe74, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:31.479+0000 I SHARDING [conn2] collection benchdb1.COL-4 was previously unsharded, new metadata loaded with shard version 1|2||54999dcc00df49298418fe74 2014-12-23T16:52:31.479+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999dcc00df49298418fe74, took 0ms 2014-12-23T16:52:31.520+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:31.520+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dcc00df49298418fe74') } 2014-12-23T16:52:32.432+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dcf9a8f659f34bfb255 2014-12-23T16:52:32.432+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:32-54999dd09a8f659f34bfb256", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353552432), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:32.499+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 1|2||54999dcc00df49298418fe74, current metadata version is 1|2||54999dcc00df49298418fe74 2014-12-23T16:52:32.500+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-4 already up to date (shard version : 1|2||54999dcc00df49298418fe74, took 0ms) 2014-12-23T16:52:32.500+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999dcc00df49298418fe74 2014-12-23T16:52:32.501+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:32.504+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:32.506+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:32.511+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:32.511+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:32.511+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:32.511+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999dcc00df49298418fe74 2014-12-23T16:52:32.574+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:32.574+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999dcc00df49298418fe74 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-4' 2014-12-23T16:52:32.880+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:32-54999dd09a8f659f34bfb257", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353552880), what: "moveChunk.commit", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:32.943+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:32.943+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:32.943+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-4 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 166 2014-12-23T16:52:32.943+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-4 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:32.943+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:33.246+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:33.246+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:33-54999dd19a8f659f34bfb258", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353553246), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 979, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 432, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:33.322+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dcc00df49298418fe74') } ntoreturn:1 keyUpdates:0 reslen:37 1801ms 2014-12-23T16:52:33.323+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:33.324+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dcc00df49298418fe74') } 2014-12-23T16:52:33.930+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dd19a8f659f34bfb259 2014-12-23T16:52:33.930+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:33-54999dd19a8f659f34bfb25a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353553930), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:33.982+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 2|0||54999dcc00df49298418fe74, current metadata version is 2|0||54999dcc00df49298418fe74 2014-12-23T16:52:33.983+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-4 from shard version 2|0||54999dcc00df49298418fe74 to shard version 2|1||54999dcc00df49298418fe74 2014-12-23T16:52:33.983+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999dcc00df49298418fe74, took 0ms 2014-12-23T16:52:33.983+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999dcc00df49298418fe74 2014-12-23T16:52:33.983+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:33.985+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:33.988+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:33.992+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:34.001+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:34.017+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:34.017+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:34.017+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:34.017+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999dcc00df49298418fe74 2014-12-23T16:52:34.094+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:34.094+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999dcc00df49298418fe74 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-4' 2014-12-23T16:52:34.399+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:34-54999dd29a8f659f34bfb25b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353554399), what: "moveChunk.commit", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:34.454+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:34.454+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:34.454+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-4 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 174 2014-12-23T16:52:34.454+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-4 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:34.454+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:34.757+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:34.757+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:34-54999dd29a8f659f34bfb25c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353554757), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 659, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 437, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:34.810+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dcc00df49298418fe74') } ntoreturn:1 keyUpdates:0 reslen:37 1487ms 2014-12-23T16:52:34.812+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dcc00df49298418fe74') } 2014-12-23T16:52:35.419+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dd29a8f659f34bfb25d 2014-12-23T16:52:35.419+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 3|0||54999dcc00df49298418fe74, current metadata version is 3|0||54999dcc00df49298418fe74 2014-12-23T16:52:35.419+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-4 from shard version 3|0||54999dcc00df49298418fe74 to shard version 3|1||54999dcc00df49298418fe74 2014-12-23T16:52:35.419+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999dcc00df49298418fe74, took 0ms 2014-12-23T16:52:35.419+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999dcc00df49298418fe74 2014-12-23T16:52:35.724+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:35-54999dd39a8f659f34bfb25e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353555724), what: "split", ns: "benchdb1.COL-4", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999dcc00df49298418fe74') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999dcc00df49298418fe74') } } } 2014-12-23T16:52:36.066+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:36.066+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dcc00df49298418fe74') } ntoreturn:1 keyUpdates:0 reslen:37 1253ms 2014-12-23T16:52:38.357+0000 I INDEX [conn2] build index on: benchdb1.COL-5 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-5" } 2014-12-23T16:52:38.357+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:38.358+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:38.658+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-5 with requested shard version 1|2||54999dd300df49298418fe79, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:38.658+0000 I SHARDING [conn2] collection benchdb1.COL-5 was previously unsharded, new metadata loaded with shard version 1|2||54999dd300df49298418fe79 2014-12-23T16:52:38.658+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999dd300df49298418fe79, took 0ms 2014-12-23T16:52:38.700+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:38.701+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dd300df49298418fe79') } 2014-12-23T16:52:39.511+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dd69a8f659f34bfb260 2014-12-23T16:52:39.511+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:39-54999dd79a8f659f34bfb261", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353559511), what: "moveChunk.start", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:39.589+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 1|2||54999dd300df49298418fe79, current metadata version is 1|2||54999dd300df49298418fe79 2014-12-23T16:52:39.590+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-5 already up to date (shard version : 1|2||54999dd300df49298418fe79, took 0ms) 2014-12-23T16:52:39.590+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999dd300df49298418fe79 2014-12-23T16:52:39.590+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:39.593+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:39.595+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:39.599+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:39.599+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:39.600+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:39.600+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999dd300df49298418fe79 2014-12-23T16:52:39.653+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:39.653+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999dd300df49298418fe79 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-5' 2014-12-23T16:52:39.925+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:39-54999dd79a8f659f34bfb262", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353559925), what: "moveChunk.commit", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:39.983+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:39.983+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:39.983+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-5 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 194 2014-12-23T16:52:39.983+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-5 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:39.983+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:40.286+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:40.286+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:40-54999dd89a8f659f34bfb263", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353560286), what: "moveChunk.from", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 889, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 383, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:40.342+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dd300df49298418fe79') } ntoreturn:1 keyUpdates:0 reslen:37 1641ms 2014-12-23T16:52:40.343+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:40.343+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dd300df49298418fe79') } 2014-12-23T16:52:40.950+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dd89a8f659f34bfb264 2014-12-23T16:52:40.950+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:40-54999dd89a8f659f34bfb265", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353560950), what: "moveChunk.start", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:41.003+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 2|0||54999dd300df49298418fe79, current metadata version is 2|0||54999dd300df49298418fe79 2014-12-23T16:52:41.004+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-5 from shard version 2|0||54999dd300df49298418fe79 to shard version 2|1||54999dd300df49298418fe79 2014-12-23T16:52:41.004+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999dd300df49298418fe79, took 0ms 2014-12-23T16:52:41.004+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999dd300df49298418fe79 2014-12-23T16:52:41.004+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:41.007+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:41.009+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:41.013+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:41.022+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:41.038+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:41.038+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:41.038+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:41.038+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999dd300df49298418fe79 2014-12-23T16:52:41.088+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:41.088+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999dd300df49298418fe79 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-5' 2014-12-23T16:52:41.392+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:41-54999dd99a8f659f34bfb266", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353561392), what: "moveChunk.commit", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:41.453+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:41.453+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:41.453+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-5 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 202 2014-12-23T16:52:41.453+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-5 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:41.454+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:41.756+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:41.756+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:41-54999dd99a8f659f34bfb267", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353561756), what: "moveChunk.from", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 660, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 415, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:41.808+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dd300df49298418fe79') } ntoreturn:1 keyUpdates:0 reslen:37 1464ms 2014-12-23T16:52:41.810+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-5", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-5-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dd300df49298418fe79') } 2014-12-23T16:52:42.417+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dd99a8f659f34bfb268 2014-12-23T16:52:42.417+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 3|0||54999dd300df49298418fe79, current metadata version is 3|0||54999dd300df49298418fe79 2014-12-23T16:52:42.418+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-5 from shard version 3|0||54999dd300df49298418fe79 to shard version 3|1||54999dd300df49298418fe79 2014-12-23T16:52:42.418+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999dd300df49298418fe79, took 0ms 2014-12-23T16:52:42.418+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999dd300df49298418fe79 2014-12-23T16:52:42.723+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:42-54999dda9a8f659f34bfb269", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353562723), what: "split", ns: "benchdb1.COL-5", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999dd300df49298418fe79') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999dd300df49298418fe79') } } } 2014-12-23T16:52:43.107+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:43.107+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-5", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-5-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dd300df49298418fe79') } ntoreturn:1 keyUpdates:0 reslen:37 1297ms 2014-12-23T16:52:45.670+0000 I INDEX [conn2] build index on: benchdb1.COL-6 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-6" } 2014-12-23T16:52:45.670+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:45.671+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:46.038+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-6 with requested shard version 1|2||54999dda00df49298418fe7c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:46.038+0000 I SHARDING [conn2] collection benchdb1.COL-6 was previously unsharded, new metadata loaded with shard version 1|2||54999dda00df49298418fe7c 2014-12-23T16:52:46.038+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999dda00df49298418fe7c, took 0ms 2014-12-23T16:52:46.083+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:46.083+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dda00df49298418fe7c') } 2014-12-23T16:52:46.906+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dde9a8f659f34bfb26b 2014-12-23T16:52:46.906+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:46-54999dde9a8f659f34bfb26c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353566906), what: "moveChunk.start", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:46.962+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 1|2||54999dda00df49298418fe7c, current metadata version is 1|2||54999dda00df49298418fe7c 2014-12-23T16:52:46.963+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-6 already up to date (shard version : 1|2||54999dda00df49298418fe7c, took 0ms) 2014-12-23T16:52:46.963+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999dda00df49298418fe7c 2014-12-23T16:52:46.964+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:46.966+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:46.969+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:46.973+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:46.973+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:46.973+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:46.973+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999dda00df49298418fe7c 2014-12-23T16:52:47.025+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:47.025+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999dda00df49298418fe7c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-6' 2014-12-23T16:52:47.242+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:47-54999ddf9a8f659f34bfb26d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353567242), what: "moveChunk.commit", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:47.309+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:47.309+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:47.309+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-6 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 221 2014-12-23T16:52:47.309+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-6 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:47.309+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:47.480+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:47.480+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:47-54999ddf9a8f659f34bfb26e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353567480), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 880, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 336, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:47.482+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('54999dda00df49298418fe7c') } 2014-12-23T16:52:47.510+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dda00df49298418fe7c') } ntoreturn:1 keyUpdates:0 reslen:37 1427ms 2014-12-23T16:52:47.511+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:47.512+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dda00df49298418fe7c') } 2014-12-23T16:52:47.814+0000 I SHARDING [conn1] could not acquire lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' (another update won) 2014-12-23T16:52:47.814+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' was not acquired. 2014-12-23T16:52:47.814+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-6 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-6 is taken. 2014-12-23T16:52:47.814+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:47-54999ddf9a8f659f34bfb271", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353567814), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-6 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey..." } } 2014-12-23T16:52:47.868+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999dda00df49298418fe7c') } ntoreturn:1 keyUpdates:0 reslen:245 356ms 2014-12-23T16:52:47.869+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dda00df49298418fe7c') } 2014-12-23T16:52:47.871+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-6 to split chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for splitting chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-6 is taken. 2014-12-23T16:52:47.873+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-6-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999dda00df49298418fe7c') } 2014-12-23T16:52:47.874+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-6 to split chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for splitting chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-6 is taken. 2014-12-23T16:52:47.880+0000 I INDEX [conn2] build index on: benchdb1.COL-7 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-7" } 2014-12-23T16:52:47.880+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:47.881+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:47.932+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ddf9a8f659f34bfb26f 2014-12-23T16:52:47.932+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:47-54999ddf9a8f659f34bfb273", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353567932), what: "moveChunk.start", ns: "benchdb1.COL-6", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:48.011+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 2|0||54999dda00df49298418fe7c, current metadata version is 2|0||54999dda00df49298418fe7c 2014-12-23T16:52:48.012+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-6 from shard version 2|0||54999dda00df49298418fe7c to shard version 2|1||54999dda00df49298418fe7c 2014-12-23T16:52:48.012+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999dda00df49298418fe7c, took 0ms 2014-12-23T16:52:48.012+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999dda00df49298418fe7c 2014-12-23T16:52:48.012+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:52:48.014+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:48.016+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:48.021+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:48.029+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:48.046+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:48.046+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:52:48.046+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:52:48.046+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999dda00df49298418fe7c 2014-12-23T16:52:48.147+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:48.147+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999dda00df49298418fe7c through { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } for collection 'benchdb1.COL-6' 2014-12-23T16:52:48.202+0000 I SHARDING [conn2] waiting till out of critical section 2014-12-23T16:52:48.405+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:48-54999de09a8f659f34bfb274", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353568405), what: "moveChunk.commit", ns: "benchdb1.COL-6", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:48.405+0000 I QUERY [conn2] command admin.$cmd command: setShardVersion { setShardVersion: "benchdb1.COL-7", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", shard: "shard0002", shardHost: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", version: Timestamp 1000|2, versionEpoch: ObjectId('54999ddc00df49298418fe80') } ntoreturn:1 keyUpdates:0 reslen:185 202ms 2014-12-23T16:52:48.406+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-7 with requested shard version 1|2||54999ddc00df49298418fe80, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:48.407+0000 I SHARDING [conn2] collection benchdb1.COL-7 was previously unsharded, new metadata loaded with shard version 1|2||54999ddc00df49298418fe80 2014-12-23T16:52:48.407+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ddc00df49298418fe80, took 0ms 2014-12-23T16:52:48.468+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:48.468+0000 I SHARDING [conn4] forking for cleanup of chunk data 2014-12-23T16:52:48.468+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:48.468+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-6 from { shardkey: MinKey } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T16:52:48.468+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-6 from { shardkey: MinKey } -> { shardkey: -3074457345618258602 } 2014-12-23T16:52:48.495+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:48.495+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ddc00df49298418fe80') } 2014-12-23T16:52:48.697+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:48.697+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:48-54999de09a8f659f34bfb275", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353568697), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 529, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 422, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:48.760+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('54999dda00df49298418fe7c') } ntoreturn:1 keyUpdates:0 reslen:37 1277ms 2014-12-23T16:52:49.308+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999de09a8f659f34bfb276 2014-12-23T16:52:49.308+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:49-54999de19a8f659f34bfb277", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353569308), what: "moveChunk.start", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:49.369+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 1|2||54999ddc00df49298418fe80, current metadata version is 1|2||54999ddc00df49298418fe80 2014-12-23T16:52:49.370+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-7 already up to date (shard version : 1|2||54999ddc00df49298418fe80, took 0ms) 2014-12-23T16:52:49.370+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999ddc00df49298418fe80 2014-12-23T16:52:49.370+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:49.373+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:49.375+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:49.379+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:49.380+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:49.380+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:49.380+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999ddc00df49298418fe80 2014-12-23T16:52:49.429+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:49.429+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999ddc00df49298418fe80 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-7' 2014-12-23T16:52:49.733+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:49-54999de19a8f659f34bfb278", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353569733), what: "moveChunk.commit", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:49.792+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:49.792+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:49.792+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-7 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 251 2014-12-23T16:52:49.792+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-7 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:49.792+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:50.078+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:50.078+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:50-54999de29a8f659f34bfb279", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353570078), what: "moveChunk.from", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 874, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 412, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:50.134+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ddc00df49298418fe80') } ntoreturn:1 keyUpdates:0 reslen:37 1639ms 2014-12-23T16:52:50.136+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:50.136+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ddc00df49298418fe80') } 2014-12-23T16:52:50.658+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999de29a8f659f34bfb27a 2014-12-23T16:52:50.658+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:50-54999de29a8f659f34bfb27b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353570658), what: "moveChunk.start", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:50.706+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 2|0||54999ddc00df49298418fe80, current metadata version is 2|0||54999ddc00df49298418fe80 2014-12-23T16:52:50.707+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-7 from shard version 2|0||54999ddc00df49298418fe80 to shard version 2|1||54999ddc00df49298418fe80 2014-12-23T16:52:50.707+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999ddc00df49298418fe80, took 0ms 2014-12-23T16:52:50.707+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999ddc00df49298418fe80 2014-12-23T16:52:50.707+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:50.709+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:50.712+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:50.716+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:50.725+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:50.741+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:50.741+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:50.741+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:50.741+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999ddc00df49298418fe80 2014-12-23T16:52:50.769+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:50.769+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999ddc00df49298418fe80 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-7' 2014-12-23T16:52:50.956+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:50-54999de29a8f659f34bfb27c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353570956), what: "moveChunk.commit", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:51.043+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:51.043+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:51.043+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-7 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 260 2014-12-23T16:52:51.043+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-7 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:51.043+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:51.247+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:51.247+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:51-54999de39a8f659f34bfb27d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353571247), what: "moveChunk.from", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 571, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 302, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:51.303+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ddc00df49298418fe80') } ntoreturn:1 keyUpdates:0 reslen:37 1167ms 2014-12-23T16:52:51.304+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-7-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ddc00df49298418fe80') } 2014-12-23T16:52:51.911+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999de39a8f659f34bfb27e 2014-12-23T16:52:51.911+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 3|0||54999ddc00df49298418fe80, current metadata version is 3|0||54999ddc00df49298418fe80 2014-12-23T16:52:51.912+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-7 from shard version 3|0||54999ddc00df49298418fe80 to shard version 3|1||54999ddc00df49298418fe80 2014-12-23T16:52:51.912+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999ddc00df49298418fe80, took 0ms 2014-12-23T16:52:51.912+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999ddc00df49298418fe80 2014-12-23T16:52:52.216+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:52-54999de49a8f659f34bfb27f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353572216), what: "split", ns: "benchdb1.COL-7", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ddc00df49298418fe80') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ddc00df49298418fe80') } } } 2014-12-23T16:52:52.597+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:52.597+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-7-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ddc00df49298418fe80') } ntoreturn:1 keyUpdates:0 reslen:37 1292ms 2014-12-23T16:52:55.202+0000 I INDEX [conn2] build index on: benchdb1.COL-8 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-8" } 2014-12-23T16:52:55.202+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:52:55.203+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:52:55.501+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-8 with requested shard version 1|2||54999de400df49298418fe86, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:52:55.502+0000 I SHARDING [conn2] collection benchdb1.COL-8 was previously unsharded, new metadata loaded with shard version 1|2||54999de400df49298418fe86 2014-12-23T16:52:55.502+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999de400df49298418fe86, took 0ms 2014-12-23T16:52:55.543+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:55.544+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999de400df49298418fe86') } 2014-12-23T16:52:56.453+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999de79a8f659f34bfb281 2014-12-23T16:52:56.453+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:56-54999de89a8f659f34bfb282", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353576453), what: "moveChunk.start", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:52:56.517+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 1|2||54999de400df49298418fe86, current metadata version is 1|2||54999de400df49298418fe86 2014-12-23T16:52:56.518+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-8 already up to date (shard version : 1|2||54999de400df49298418fe86, took 0ms) 2014-12-23T16:52:56.518+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999de400df49298418fe86 2014-12-23T16:52:56.518+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:56.521+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:56.524+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:56.528+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:56.528+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:56.528+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:56.528+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999de400df49298418fe86 2014-12-23T16:52:56.578+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:56.578+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999de400df49298418fe86 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-8' 2014-12-23T16:52:56.882+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:56-54999de89a8f659f34bfb283", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353576882), what: "moveChunk.commit", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:56.942+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:56.942+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:56.942+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-8 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 279 2014-12-23T16:52:56.942+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-8 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:52:56.942+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:57.245+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:57.245+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:57-54999de99a8f659f34bfb284", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353577245), what: "moveChunk.from", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 974, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:52:57.292+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999de400df49298418fe86') } ntoreturn:1 keyUpdates:0 reslen:37 1748ms 2014-12-23T16:52:57.293+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:52:57.294+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999de400df49298418fe86') } 2014-12-23T16:52:57.900+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999de99a8f659f34bfb285 2014-12-23T16:52:57.900+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:57-54999de99a8f659f34bfb286", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353577900), what: "moveChunk.start", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:52:57.954+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 2|0||54999de400df49298418fe86, current metadata version is 2|0||54999de400df49298418fe86 2014-12-23T16:52:57.955+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-8 from shard version 2|0||54999de400df49298418fe86 to shard version 2|1||54999de400df49298418fe86 2014-12-23T16:52:57.955+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999de400df49298418fe86, took 0ms 2014-12-23T16:52:57.955+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999de400df49298418fe86 2014-12-23T16:52:57.955+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:52:57.957+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:57.960+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:57.964+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:57.973+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:57.989+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:52:57.989+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:52:57.989+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:52:57.989+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999de400df49298418fe86 2014-12-23T16:52:58.040+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:52:58.040+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999de400df49298418fe86 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-8' 2014-12-23T16:52:58.344+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:58-54999dea9a8f659f34bfb287", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353578344), what: "moveChunk.commit", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:52:58.403+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:58.403+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:52:58.403+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-8 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 287 2014-12-23T16:52:58.403+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-8 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:52:58.403+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:52:58.706+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:52:58.706+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:58-54999dea9a8f659f34bfb288", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353578706), what: "moveChunk.from", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 661, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 413, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:52:58.756+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999de400df49298418fe86') } ntoreturn:1 keyUpdates:0 reslen:37 1463ms 2014-12-23T16:52:58.758+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-8-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999de400df49298418fe86') } 2014-12-23T16:52:59.364+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dea9a8f659f34bfb289 2014-12-23T16:52:59.364+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 3|0||54999de400df49298418fe86, current metadata version is 3|0||54999de400df49298418fe86 2014-12-23T16:52:59.365+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-8 from shard version 3|0||54999de400df49298418fe86 to shard version 3|1||54999de400df49298418fe86 2014-12-23T16:52:59.365+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999de400df49298418fe86, took 0ms 2014-12-23T16:52:59.365+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999de400df49298418fe86 2014-12-23T16:52:59.647+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:52:59-54999deb9a8f659f34bfb28a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353579647), what: "split", ns: "benchdb1.COL-8", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999de400df49298418fe86') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999de400df49298418fe86') } } } 2014-12-23T16:53:00.031+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:00.031+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-8-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999de400df49298418fe86') } ntoreturn:1 keyUpdates:0 reslen:37 1273ms 2014-12-23T16:53:01.854+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:53:01 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:53:02.170+0000 I INDEX [conn2] build index on: benchdb1.COL-9 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-9" } 2014-12-23T16:53:02.170+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:02.171+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:02.508+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-9 with requested shard version 1|2||54999deb00df49298418fe8b, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:02.509+0000 I SHARDING [conn2] collection benchdb1.COL-9 was previously unsharded, new metadata loaded with shard version 1|2||54999deb00df49298418fe8b 2014-12-23T16:53:02.509+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999deb00df49298418fe8b, took 0ms 2014-12-23T16:53:02.548+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:02.549+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999deb00df49298418fe8b') } 2014-12-23T16:53:03.453+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dee9a8f659f34bfb28c 2014-12-23T16:53:03.453+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:03-54999def9a8f659f34bfb28d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353583453), what: "moveChunk.start", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:03.526+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 1|2||54999deb00df49298418fe8b, current metadata version is 1|2||54999deb00df49298418fe8b 2014-12-23T16:53:03.527+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-9 already up to date (shard version : 1|2||54999deb00df49298418fe8b, took 0ms) 2014-12-23T16:53:03.527+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999deb00df49298418fe8b 2014-12-23T16:53:03.527+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:03.530+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:03.532+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:03.536+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:03.536+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:03.537+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:03.537+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999deb00df49298418fe8b 2014-12-23T16:53:03.590+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:03.590+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999deb00df49298418fe8b through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-9' 2014-12-23T16:53:03.896+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:03-54999def9a8f659f34bfb28e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353583896), what: "moveChunk.commit", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:03.954+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:03.954+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:03.954+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-9 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 308 2014-12-23T16:53:03.954+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-9 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:03.954+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:04.258+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:04.258+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:04-54999df09a8f659f34bfb28f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353584258), what: "moveChunk.from", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 978, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 417, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:04.308+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999deb00df49298418fe8b') } ntoreturn:1 keyUpdates:0 reslen:37 1760ms 2014-12-23T16:53:04.310+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:04.310+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999deb00df49298418fe8b') } 2014-12-23T16:53:04.918+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999df09a8f659f34bfb290 2014-12-23T16:53:04.918+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:04-54999df09a8f659f34bfb291", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353584918), what: "moveChunk.start", ns: "benchdb1.COL-9", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:04.973+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 2|0||54999deb00df49298418fe8b, current metadata version is 2|0||54999deb00df49298418fe8b 2014-12-23T16:53:04.974+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-9 from shard version 2|0||54999deb00df49298418fe8b to shard version 2|1||54999deb00df49298418fe8b 2014-12-23T16:53:04.974+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999deb00df49298418fe8b, took 0ms 2014-12-23T16:53:04.974+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999deb00df49298418fe8b 2014-12-23T16:53:04.974+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:04.976+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:04.979+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:04.983+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:04.992+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:05.008+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:05.008+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:05.009+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:05.009+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999deb00df49298418fe8b 2014-12-23T16:53:05.059+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:05.059+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999deb00df49298418fe8b through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-9' 2014-12-23T16:53:05.364+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:05-54999df19a8f659f34bfb292", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353585364), what: "moveChunk.commit", ns: "benchdb1.COL-9", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:05.456+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:05.457+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:05.457+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-9 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 316 2014-12-23T16:53:05.457+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-9 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:05.457+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:05.760+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:05.760+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:05-54999df19a8f659f34bfb293", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353585760), what: "moveChunk.from", ns: "benchdb1.COL-9", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 448, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:05.808+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999deb00df49298418fe8b') } ntoreturn:1 keyUpdates:0 reslen:37 1497ms 2014-12-23T16:53:05.809+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-9-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999deb00df49298418fe8b') } 2014-12-23T16:53:06.416+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999df19a8f659f34bfb294 2014-12-23T16:53:06.416+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 3|0||54999deb00df49298418fe8b, current metadata version is 3|0||54999deb00df49298418fe8b 2014-12-23T16:53:06.417+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-9 from shard version 3|0||54999deb00df49298418fe8b to shard version 3|1||54999deb00df49298418fe8b 2014-12-23T16:53:06.417+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999deb00df49298418fe8b, took 0ms 2014-12-23T16:53:06.417+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999deb00df49298418fe8b 2014-12-23T16:53:06.722+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:06-54999df29a8f659f34bfb295", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353586722), what: "split", ns: "benchdb1.COL-9", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999deb00df49298418fe8b') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999deb00df49298418fe8b') } } } 2014-12-23T16:53:07.081+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:07.081+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-9-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999deb00df49298418fe8b') } ntoreturn:1 keyUpdates:0 reslen:37 1271ms 2014-12-23T16:53:09.633+0000 I INDEX [conn2] build index on: benchdb1.COL-10 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-10" } 2014-12-23T16:53:09.633+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:09.634+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:09.935+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 1|2||54999df200df49298418fe8e, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:09.936+0000 I SHARDING [conn2] collection benchdb1.COL-10 was previously unsharded, new metadata loaded with shard version 1|2||54999df200df49298418fe8e 2014-12-23T16:53:09.936+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999df200df49298418fe8e, took 0ms 2014-12-23T16:53:09.977+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:09.978+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df200df49298418fe8e') } 2014-12-23T16:53:10.848+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999df69a8f659f34bfb297 2014-12-23T16:53:10.848+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:10-54999df69a8f659f34bfb298", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353590848), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:10.902+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 1|2||54999df200df49298418fe8e, current metadata version is 1|2||54999df200df49298418fe8e 2014-12-23T16:53:10.903+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-10 already up to date (shard version : 1|2||54999df200df49298418fe8e, took 0ms) 2014-12-23T16:53:10.903+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999df200df49298418fe8e 2014-12-23T16:53:10.903+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:10.906+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:10.908+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:10.913+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:10.913+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:10.913+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:10.913+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999df200df49298418fe8e 2014-12-23T16:53:10.966+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:10.966+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999df200df49298418fe8e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T16:53:11.271+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:11-54999df79a8f659f34bfb299", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353591271), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:11.326+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:11.326+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:11.326+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 335 2014-12-23T16:53:11.326+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:11.326+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:11.629+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:11.629+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:11-54999df79a8f659f34bfb29a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353591629), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 924, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 413, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:11.681+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df200df49298418fe8e') } ntoreturn:1 keyUpdates:0 reslen:37 1704ms 2014-12-23T16:53:11.683+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:11.683+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df200df49298418fe8e') } 2014-12-23T16:53:12.292+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999df79a8f659f34bfb29b 2014-12-23T16:53:12.292+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:12-54999df89a8f659f34bfb29c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353592292), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:12.345+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 2|0||54999df200df49298418fe8e, current metadata version is 2|0||54999df200df49298418fe8e 2014-12-23T16:53:12.346+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-10 from shard version 2|0||54999df200df49298418fe8e to shard version 2|1||54999df200df49298418fe8e 2014-12-23T16:53:12.346+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999df200df49298418fe8e, took 0ms 2014-12-23T16:53:12.346+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999df200df49298418fe8e 2014-12-23T16:53:12.346+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:12.348+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:12.351+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:12.355+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:12.364+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:12.380+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:12.380+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:12.381+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:12.381+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999df200df49298418fe8e 2014-12-23T16:53:12.435+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:12.435+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999df200df49298418fe8e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T16:53:12.740+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:12-54999df89a8f659f34bfb29d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353592740), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:12.827+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:12.827+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:12.827+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 343 2014-12-23T16:53:12.827+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:12.827+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:13.025+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:13.025+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:13-54999df99a8f659f34bfb29e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353593025), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 446, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:13.053+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df200df49298418fe8e') } ntoreturn:1 keyUpdates:0 reslen:37 1370ms 2014-12-23T16:53:13.055+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999df200df49298418fe8e') } 2014-12-23T16:53:13.343+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999df99a8f659f34bfb29f 2014-12-23T16:53:13.343+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 3|0||54999df200df49298418fe8e, current metadata version is 3|0||54999df200df49298418fe8e 2014-12-23T16:53:13.344+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-10 from shard version 3|0||54999df200df49298418fe8e to shard version 3|1||54999df200df49298418fe8e 2014-12-23T16:53:13.344+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999df200df49298418fe8e, took 0ms 2014-12-23T16:53:13.344+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999df200df49298418fe8e 2014-12-23T16:53:13.606+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:13-54999df99a8f659f34bfb2a0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353593606), what: "split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999df200df49298418fe8e') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999df200df49298418fe8e') } } } 2014-12-23T16:53:13.988+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:13.988+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999df200df49298418fe8e') } ntoreturn:1 keyUpdates:0 reslen:37 932ms 2014-12-23T16:53:16.586+0000 I INDEX [conn2] build index on: benchdb1.COL-11 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-11" } 2014-12-23T16:53:16.586+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:16.587+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:16.916+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 1|2||54999df900df49298418fe93, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:16.917+0000 I SHARDING [conn2] collection benchdb1.COL-11 was previously unsharded, new metadata loaded with shard version 1|2||54999df900df49298418fe93 2014-12-23T16:53:16.917+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999df900df49298418fe93, took 0ms 2014-12-23T16:53:16.958+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:16.958+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df900df49298418fe93') } 2014-12-23T16:53:17.868+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dfd9a8f659f34bfb2a2 2014-12-23T16:53:17.868+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:17-54999dfd9a8f659f34bfb2a3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353597868), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:17.925+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 1|2||54999df900df49298418fe93, current metadata version is 1|2||54999df900df49298418fe93 2014-12-23T16:53:17.926+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-11 already up to date (shard version : 1|2||54999df900df49298418fe93, took 0ms) 2014-12-23T16:53:17.926+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999df900df49298418fe93 2014-12-23T16:53:17.927+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:17.929+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:17.931+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:17.936+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:17.936+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:17.936+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:17.936+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999df900df49298418fe93 2014-12-23T16:53:17.985+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:17.985+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999df900df49298418fe93 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T16:53:18.290+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:18-54999dfe9a8f659f34bfb2a4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353598290), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:18.352+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:18.352+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:18.352+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 364 2014-12-23T16:53:18.352+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:18.352+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:18.655+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:18.655+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:18-54999dfe9a8f659f34bfb2a5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353598655), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 967, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 415, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:18.701+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df900df49298418fe93') } ntoreturn:1 keyUpdates:0 reslen:37 1743ms 2014-12-23T16:53:18.702+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:18.703+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df900df49298418fe93') } 2014-12-23T16:53:19.309+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999dfe9a8f659f34bfb2a6 2014-12-23T16:53:19.309+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:19-54999dff9a8f659f34bfb2a7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353599309), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:19.362+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 2|0||54999df900df49298418fe93, current metadata version is 2|0||54999df900df49298418fe93 2014-12-23T16:53:19.363+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-11 from shard version 2|0||54999df900df49298418fe93 to shard version 2|1||54999df900df49298418fe93 2014-12-23T16:53:19.363+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999df900df49298418fe93, took 0ms 2014-12-23T16:53:19.363+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999df900df49298418fe93 2014-12-23T16:53:19.363+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:19.365+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:19.368+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:19.372+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:19.380+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:19.397+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:19.397+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:19.397+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:19.397+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999df900df49298418fe93 2014-12-23T16:53:19.444+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:19.444+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999df900df49298418fe93 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T16:53:19.748+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:19-54999dff9a8f659f34bfb2a8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353599748), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:19.806+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:19.806+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:19.806+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 372 2014-12-23T16:53:19.806+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:19.806+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:20.081+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:20.081+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:20-54999e009a8f659f34bfb2a9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353600081), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 660, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 408, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:20.126+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999df900df49298418fe93') } ntoreturn:1 keyUpdates:0 reslen:37 1423ms 2014-12-23T16:53:20.128+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999df900df49298418fe93') } 2014-12-23T16:53:20.697+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e009a8f659f34bfb2aa 2014-12-23T16:53:20.697+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 3|0||54999df900df49298418fe93, current metadata version is 3|0||54999df900df49298418fe93 2014-12-23T16:53:20.698+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-11 from shard version 3|0||54999df900df49298418fe93 to shard version 3|1||54999df900df49298418fe93 2014-12-23T16:53:20.698+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999df900df49298418fe93, took 0ms 2014-12-23T16:53:20.698+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999df900df49298418fe93 2014-12-23T16:53:21.002+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:21-54999e019a8f659f34bfb2ab", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353601002), what: "split", ns: "benchdb1.COL-11", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999df900df49298418fe93') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999df900df49298418fe93') } } } 2014-12-23T16:53:21.388+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:21.388+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999df900df49298418fe93') } ntoreturn:1 keyUpdates:0 reslen:37 1259ms 2014-12-23T16:53:23.905+0000 I INDEX [conn2] build index on: benchdb1.COL-12 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-12" } 2014-12-23T16:53:23.905+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:23.906+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:24.241+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 1|2||54999e0000df49298418fe97, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:24.242+0000 I SHARDING [conn2] collection benchdb1.COL-12 was previously unsharded, new metadata loaded with shard version 1|2||54999e0000df49298418fe97 2014-12-23T16:53:24.242+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e0000df49298418fe97, took 0ms 2014-12-23T16:53:24.286+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:24.286+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0000df49298418fe97') } 2014-12-23T16:53:24.691+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e049a8f659f34bfb2ad 2014-12-23T16:53:24.691+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:24-54999e049a8f659f34bfb2ae", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353604691), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:24.734+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 1|2||54999e0000df49298418fe97, current metadata version is 1|2||54999e0000df49298418fe97 2014-12-23T16:53:24.735+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-12 already up to date (shard version : 1|2||54999e0000df49298418fe97, took 0ms) 2014-12-23T16:53:24.735+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e0000df49298418fe97 2014-12-23T16:53:24.736+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:24.738+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:24.741+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:24.745+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:24.745+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:24.745+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:24.745+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e0000df49298418fe97 2014-12-23T16:53:24.797+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:24.797+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e0000df49298418fe97 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T16:53:25.101+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:25-54999e059a8f659f34bfb2af", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353605101), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:25.169+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:25.169+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:25.169+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-12 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 393 2014-12-23T16:53:25.169+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:25.169+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:25.472+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:25.472+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:25-54999e059a8f659f34bfb2b0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353605472), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 449, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 423, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:25.521+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0000df49298418fe97') } ntoreturn:1 keyUpdates:0 reslen:37 1235ms 2014-12-23T16:53:25.522+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:25.523+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0000df49298418fe97') } 2014-12-23T16:53:26.129+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e059a8f659f34bfb2b1 2014-12-23T16:53:26.129+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:26-54999e069a8f659f34bfb2b2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353606129), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:26.184+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 2|0||54999e0000df49298418fe97, current metadata version is 2|0||54999e0000df49298418fe97 2014-12-23T16:53:26.184+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-12 from shard version 2|0||54999e0000df49298418fe97 to shard version 2|1||54999e0000df49298418fe97 2014-12-23T16:53:26.184+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e0000df49298418fe97, took 0ms 2014-12-23T16:53:26.184+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e0000df49298418fe97 2014-12-23T16:53:26.185+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:26.187+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:26.189+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:26.193+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:26.202+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:26.218+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:26.218+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:26.219+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:26.219+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e0000df49298418fe97 2014-12-23T16:53:26.267+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:26.267+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e0000df49298418fe97 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T16:53:26.571+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:26-54999e069a8f659f34bfb2b3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353606571), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:26.626+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:26.626+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:26.626+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 401 2014-12-23T16:53:26.626+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:26.626+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:26.929+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:26.929+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:26-54999e069a8f659f34bfb2b4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353606929), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 661, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 407, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:26.979+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0000df49298418fe97') } ntoreturn:1 keyUpdates:0 reslen:37 1457ms 2014-12-23T16:53:26.981+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e0000df49298418fe97') } 2014-12-23T16:53:27.588+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e069a8f659f34bfb2b5 2014-12-23T16:53:27.588+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 3|0||54999e0000df49298418fe97, current metadata version is 3|0||54999e0000df49298418fe97 2014-12-23T16:53:27.588+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-12 from shard version 3|0||54999e0000df49298418fe97 to shard version 3|1||54999e0000df49298418fe97 2014-12-23T16:53:27.588+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e0000df49298418fe97, took 0ms 2014-12-23T16:53:27.588+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e0000df49298418fe97 2014-12-23T16:53:27.893+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:27-54999e079a8f659f34bfb2b6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353607893), what: "split", ns: "benchdb1.COL-12", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e0000df49298418fe97') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e0000df49298418fe97') } } } 2014-12-23T16:53:28.269+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:28.269+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e0000df49298418fe97') } ntoreturn:1 keyUpdates:0 reslen:37 1287ms 2014-12-23T16:53:30.857+0000 I INDEX [conn2] build index on: benchdb1.COL-13 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-13" } 2014-12-23T16:53:30.857+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:30.858+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:31.145+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 1|2||54999e0700df49298418fe9b, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:31.146+0000 I SHARDING [conn2] collection benchdb1.COL-13 was previously unsharded, new metadata loaded with shard version 1|2||54999e0700df49298418fe9b 2014-12-23T16:53:31.146+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e0700df49298418fe9b, took 0ms 2014-12-23T16:53:31.191+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:31.192+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0700df49298418fe9b') } 2014-12-23T16:53:32.052+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e0b9a8f659f34bfb2b8 2014-12-23T16:53:32.052+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:32-54999e0c9a8f659f34bfb2b9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353612052), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:32.117+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 1|2||54999e0700df49298418fe9b, current metadata version is 1|2||54999e0700df49298418fe9b 2014-12-23T16:53:32.117+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-13 already up to date (shard version : 1|2||54999e0700df49298418fe9b, took 0ms) 2014-12-23T16:53:32.117+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e0700df49298418fe9b 2014-12-23T16:53:32.118+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:32.120+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:32.123+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:32.127+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:32.127+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:32.127+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:32.127+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e0700df49298418fe9b 2014-12-23T16:53:32.175+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:32.175+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e0700df49298418fe9b through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T16:53:32.326+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:53:31 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:53:32.479+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:32-54999e0c9a8f659f34bfb2ba", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353612479), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:32.539+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:32.539+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:32.539+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 420 2014-12-23T16:53:32.539+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:32.539+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:32.843+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:32.843+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:32-54999e0c9a8f659f34bfb2bb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353612843), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 925, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 411, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:32.890+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0700df49298418fe9b') } ntoreturn:1 keyUpdates:0 reslen:37 1699ms 2014-12-23T16:53:32.892+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:32.892+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0700df49298418fe9b') } 2014-12-23T16:53:33.499+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e0c9a8f659f34bfb2bc 2014-12-23T16:53:33.499+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:33-54999e0d9a8f659f34bfb2bd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353613499), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:33.560+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 2|0||54999e0700df49298418fe9b, current metadata version is 2|0||54999e0700df49298418fe9b 2014-12-23T16:53:33.561+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-13 from shard version 2|0||54999e0700df49298418fe9b to shard version 2|1||54999e0700df49298418fe9b 2014-12-23T16:53:33.561+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e0700df49298418fe9b, took 0ms 2014-12-23T16:53:33.561+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e0700df49298418fe9b 2014-12-23T16:53:33.561+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:33.563+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:33.566+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:33.570+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:33.579+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:33.595+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:33.595+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:33.595+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:33.596+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e0700df49298418fe9b 2014-12-23T16:53:33.643+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:33.643+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e0700df49298418fe9b through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T16:53:33.949+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:33-54999e0d9a8f659f34bfb2be", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353613949), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:34.023+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:34.023+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:34.023+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 428 2014-12-23T16:53:34.023+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:34.023+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:34.326+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:34.326+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:34-54999e0e9a8f659f34bfb2bf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353614326), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 427, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:34.372+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0700df49298418fe9b') } ntoreturn:1 keyUpdates:0 reslen:37 1480ms 2014-12-23T16:53:34.374+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e0700df49298418fe9b') } 2014-12-23T16:53:34.951+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e0e9a8f659f34bfb2c0 2014-12-23T16:53:34.951+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 3|0||54999e0700df49298418fe9b, current metadata version is 3|0||54999e0700df49298418fe9b 2014-12-23T16:53:34.952+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-13 from shard version 3|0||54999e0700df49298418fe9b to shard version 3|1||54999e0700df49298418fe9b 2014-12-23T16:53:34.952+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e0700df49298418fe9b, took 0ms 2014-12-23T16:53:34.952+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e0700df49298418fe9b 2014-12-23T16:53:35.208+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:35-54999e0f9a8f659f34bfb2c1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353615208), what: "split", ns: "benchdb1.COL-13", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e0700df49298418fe9b') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e0700df49298418fe9b') } } } 2014-12-23T16:53:35.566+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:35.566+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e0700df49298418fe9b') } ntoreturn:1 keyUpdates:0 reslen:37 1191ms 2014-12-23T16:53:37.595+0000 I INDEX [conn2] build index on: benchdb1.COL-14 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-14" } 2014-12-23T16:53:37.595+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:37.596+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:37.859+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 1|2||54999e0e00df49298418fea0, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:37.859+0000 I SHARDING [conn2] collection benchdb1.COL-14 was previously unsharded, new metadata loaded with shard version 1|2||54999e0e00df49298418fea0 2014-12-23T16:53:37.859+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e0e00df49298418fea0, took 0ms 2014-12-23T16:53:37.913+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:37.914+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0e00df49298418fea0') } 2014-12-23T16:53:38.823+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e129a8f659f34bfb2c3 2014-12-23T16:53:38.823+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:38-54999e129a8f659f34bfb2c4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353618823), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:38.889+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 1|2||54999e0e00df49298418fea0, current metadata version is 1|2||54999e0e00df49298418fea0 2014-12-23T16:53:38.890+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-14 already up to date (shard version : 1|2||54999e0e00df49298418fea0, took 0ms) 2014-12-23T16:53:38.890+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e0e00df49298418fea0 2014-12-23T16:53:38.890+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:38.893+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:38.895+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:38.900+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:38.900+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:38.900+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:38.900+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e0e00df49298418fea0 2014-12-23T16:53:38.953+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:38.953+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e0e00df49298418fea0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T16:53:39.257+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:39-54999e139a8f659f34bfb2c5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353619257), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:39.330+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:39.330+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:39.330+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 449 2014-12-23T16:53:39.330+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:39.330+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:39.633+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:39.633+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:39-54999e139a8f659f34bfb2c6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353619633), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 976, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 429, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:39.684+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0e00df49298418fea0') } ntoreturn:1 keyUpdates:0 reslen:37 1770ms 2014-12-23T16:53:39.686+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:39.686+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0e00df49298418fea0') } 2014-12-23T16:53:40.194+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e139a8f659f34bfb2c7 2014-12-23T16:53:40.194+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:40-54999e149a8f659f34bfb2c8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353620194), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:40.249+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 2|0||54999e0e00df49298418fea0, current metadata version is 2|0||54999e0e00df49298418fea0 2014-12-23T16:53:40.250+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-14 from shard version 2|0||54999e0e00df49298418fea0 to shard version 2|1||54999e0e00df49298418fea0 2014-12-23T16:53:40.250+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e0e00df49298418fea0, took 0ms 2014-12-23T16:53:40.250+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e0e00df49298418fea0 2014-12-23T16:53:40.250+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:40.252+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:40.255+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:40.259+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:40.267+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:40.284+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:40.284+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:40.284+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:40.284+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e0e00df49298418fea0 2014-12-23T16:53:40.335+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:40.335+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e0e00df49298418fea0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T16:53:40.639+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:40-54999e149a8f659f34bfb2c9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353620639), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:40.699+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:40.699+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:40.699+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 457 2014-12-23T16:53:40.699+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:40.699+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:41.002+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:41.002+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:41-54999e159a8f659f34bfb2ca", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353621002), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 563, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 414, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:41.050+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e0e00df49298418fea0') } ntoreturn:1 keyUpdates:0 reslen:37 1364ms 2014-12-23T16:53:41.052+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e0e00df49298418fea0') } 2014-12-23T16:53:41.658+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e159a8f659f34bfb2cb 2014-12-23T16:53:41.658+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 3|0||54999e0e00df49298418fea0, current metadata version is 3|0||54999e0e00df49298418fea0 2014-12-23T16:53:41.659+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-14 from shard version 3|0||54999e0e00df49298418fea0 to shard version 3|1||54999e0e00df49298418fea0 2014-12-23T16:53:41.659+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e0e00df49298418fea0, took 0ms 2014-12-23T16:53:41.659+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e0e00df49298418fea0 2014-12-23T16:53:41.963+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:41-54999e159a8f659f34bfb2cc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353621963), what: "split", ns: "benchdb1.COL-14", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e0e00df49298418fea0') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e0e00df49298418fea0') } } } 2014-12-23T16:53:42.351+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:42.351+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e0e00df49298418fea0') } ntoreturn:1 keyUpdates:0 reslen:37 1299ms 2014-12-23T16:53:44.986+0000 I INDEX [conn2] build index on: benchdb1.COL-15 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-15" } 2014-12-23T16:53:44.986+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:44.987+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:45.286+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 1|2||54999e1600df49298418fea3, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:45.287+0000 I SHARDING [conn2] collection benchdb1.COL-15 was previously unsharded, new metadata loaded with shard version 1|2||54999e1600df49298418fea3 2014-12-23T16:53:45.287+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e1600df49298418fea3, took 0ms 2014-12-23T16:53:45.329+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:45.329+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1600df49298418fea3') } 2014-12-23T16:53:46.239+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e199a8f659f34bfb2ce 2014-12-23T16:53:46.239+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:46-54999e1a9a8f659f34bfb2cf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353626239), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:46.304+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 1|2||54999e1600df49298418fea3, current metadata version is 1|2||54999e1600df49298418fea3 2014-12-23T16:53:46.304+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-15 already up to date (shard version : 1|2||54999e1600df49298418fea3, took 0ms) 2014-12-23T16:53:46.304+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e1600df49298418fea3 2014-12-23T16:53:46.305+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:46.307+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:46.310+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:46.314+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:46.314+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:46.314+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:46.314+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e1600df49298418fea3 2014-12-23T16:53:46.367+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:46.367+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e1600df49298418fea3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T16:53:46.668+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:46-54999e1a9a8f659f34bfb2d0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353626668), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:46.729+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:46.730+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:46.730+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 476 2014-12-23T16:53:46.730+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:46.730+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:46.884+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:46.884+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:46-54999e1a9a8f659f34bfb2d1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353626884), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 974, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 415, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:46.913+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1600df49298418fea3') } ntoreturn:1 keyUpdates:0 reslen:37 1583ms 2014-12-23T16:53:46.914+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:46.915+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1600df49298418fea3') } 2014-12-23T16:53:47.199+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e1a9a8f659f34bfb2d2 2014-12-23T16:53:47.199+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:47-54999e1b9a8f659f34bfb2d3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353627199), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:47.251+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 2|0||54999e1600df49298418fea3, current metadata version is 2|0||54999e1600df49298418fea3 2014-12-23T16:53:47.251+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-15 from shard version 2|0||54999e1600df49298418fea3 to shard version 2|1||54999e1600df49298418fea3 2014-12-23T16:53:47.251+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e1600df49298418fea3, took 0ms 2014-12-23T16:53:47.251+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e1600df49298418fea3 2014-12-23T16:53:47.251+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:47.254+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:47.256+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:47.260+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:47.269+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:47.283+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('54999e1600df49298418fea3') } 2014-12-23T16:53:47.283+0000 W SHARDING [conn1] migration already in progress 2014-12-23T16:53:47.283+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:47-54999e1b9a8f659f34bfb2d4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353627283), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T16:53:47.285+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:47.285+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:47.286+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:47.286+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e1600df49298418fea3 2014-12-23T16:53:47.374+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:47.374+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e1600df49298418fea3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T16:53:47.655+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:47-54999e1b9a8f659f34bfb2d5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353627655), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:47.726+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:47.726+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:47.726+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 486 2014-12-23T16:53:47.726+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:47.726+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:48.029+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:48.029+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:48-54999e1c9a8f659f34bfb2d6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353628029), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 336, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 440, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:48.082+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1600df49298418fea3') } ntoreturn:1 keyUpdates:0 reslen:37 1167ms 2014-12-23T16:53:48.083+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e1600df49298418fea3') } 2014-12-23T16:53:48.689+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e1c9a8f659f34bfb2d7 2014-12-23T16:53:48.690+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 3|0||54999e1600df49298418fea3, current metadata version is 3|0||54999e1600df49298418fea3 2014-12-23T16:53:48.690+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-15 from shard version 3|0||54999e1600df49298418fea3 to shard version 3|1||54999e1600df49298418fea3 2014-12-23T16:53:48.690+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e1600df49298418fea3, took 0ms 2014-12-23T16:53:48.690+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e1600df49298418fea3 2014-12-23T16:53:48.994+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:48-54999e1c9a8f659f34bfb2d8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353628994), what: "split", ns: "benchdb1.COL-15", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e1600df49298418fea3') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e1600df49298418fea3') } } } 2014-12-23T16:53:49.368+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:49.368+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e1600df49298418fea3') } ntoreturn:1 keyUpdates:0 reslen:37 1284ms 2014-12-23T16:53:51.947+0000 I INDEX [conn2] build index on: benchdb1.COL-16 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-16" } 2014-12-23T16:53:51.947+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:51.948+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:52.283+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 1|2||54999e1c00df49298418fea8, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:52.284+0000 I SHARDING [conn2] collection benchdb1.COL-16 was previously unsharded, new metadata loaded with shard version 1|2||54999e1c00df49298418fea8 2014-12-23T16:53:52.284+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e1c00df49298418fea8, took 0ms 2014-12-23T16:53:52.323+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:52.324+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1c00df49298418fea8') } 2014-12-23T16:53:53.233+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e209a8f659f34bfb2da 2014-12-23T16:53:53.233+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:53-54999e219a8f659f34bfb2db", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353633233), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:53.289+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 1|2||54999e1c00df49298418fea8, current metadata version is 1|2||54999e1c00df49298418fea8 2014-12-23T16:53:53.290+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-16 already up to date (shard version : 1|2||54999e1c00df49298418fea8, took 0ms) 2014-12-23T16:53:53.290+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e1c00df49298418fea8 2014-12-23T16:53:53.291+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:53.293+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:53.296+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:53.300+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:53.300+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:53.300+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:53.300+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e1c00df49298418fea8 2014-12-23T16:53:53.349+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:53.349+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e1c00df49298418fea8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T16:53:53.654+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:53-54999e219a8f659f34bfb2dc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353633654), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:53.715+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:53.715+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:53.715+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 506 2014-12-23T16:53:53.715+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:53:53.715+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:54.018+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:54.018+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:54-54999e229a8f659f34bfb2dd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353634018), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 966, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:53:54.066+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1c00df49298418fea8') } ntoreturn:1 keyUpdates:0 reslen:37 1742ms 2014-12-23T16:53:54.068+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:54.068+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1c00df49298418fea8') } 2014-12-23T16:53:54.674+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e229a8f659f34bfb2de 2014-12-23T16:53:54.674+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:54-54999e229a8f659f34bfb2df", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353634674), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:53:54.728+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 2|0||54999e1c00df49298418fea8, current metadata version is 2|0||54999e1c00df49298418fea8 2014-12-23T16:53:54.728+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-16 from shard version 2|0||54999e1c00df49298418fea8 to shard version 2|1||54999e1c00df49298418fea8 2014-12-23T16:53:54.728+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e1c00df49298418fea8, took 0ms 2014-12-23T16:53:54.728+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e1c00df49298418fea8 2014-12-23T16:53:54.728+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:54.731+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:54.733+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:54.737+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:54.746+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:54.762+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:54.762+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:53:54.763+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:53:54.763+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e1c00df49298418fea8 2014-12-23T16:53:54.809+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:53:54.809+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e1c00df49298418fea8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T16:53:55.113+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:55-54999e239a8f659f34bfb2e0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353635113), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:53:55.171+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:55.171+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:53:55.171+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 514 2014-12-23T16:53:55.171+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:53:55.171+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:53:55.474+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:55.474+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:55-54999e239a8f659f34bfb2e1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353635474), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 660, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 408, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:53:55.522+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e1c00df49298418fea8') } ntoreturn:1 keyUpdates:0 reslen:37 1453ms 2014-12-23T16:53:55.523+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e1c00df49298418fea8') } 2014-12-23T16:53:56.129+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e239a8f659f34bfb2e2 2014-12-23T16:53:56.129+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 3|0||54999e1c00df49298418fea8, current metadata version is 3|0||54999e1c00df49298418fea8 2014-12-23T16:53:56.130+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-16 from shard version 3|0||54999e1c00df49298418fea8 to shard version 3|1||54999e1c00df49298418fea8 2014-12-23T16:53:56.130+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e1c00df49298418fea8, took 0ms 2014-12-23T16:53:56.130+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e1c00df49298418fea8 2014-12-23T16:53:56.434+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:56-54999e249a8f659f34bfb2e3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353636434), what: "split", ns: "benchdb1.COL-16", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e1c00df49298418fea8') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e1c00df49298418fea8') } } } 2014-12-23T16:53:56.817+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:53:56.817+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e1c00df49298418fea8') } ntoreturn:1 keyUpdates:0 reslen:37 1293ms 2014-12-23T16:53:58.784+0000 I INDEX [conn2] build index on: benchdb1.COL-17 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-17" } 2014-12-23T16:53:58.784+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:53:58.785+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:53:59.070+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-17 with requested shard version 1|2||54999e2300df49298418feac, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:53:59.071+0000 I SHARDING [conn2] collection benchdb1.COL-17 was previously unsharded, new metadata loaded with shard version 1|2||54999e2300df49298418feac 2014-12-23T16:53:59.071+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e2300df49298418feac, took 0ms 2014-12-23T16:53:59.144+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:53:59.145+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2300df49298418feac') } 2014-12-23T16:53:59.934+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e279a8f659f34bfb2e5 2014-12-23T16:53:59.934+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:53:59-54999e279a8f659f34bfb2e6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353639934), what: "moveChunk.start", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:53:59.992+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 1|2||54999e2300df49298418feac, current metadata version is 1|2||54999e2300df49298418feac 2014-12-23T16:53:59.993+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-17 already up to date (shard version : 1|2||54999e2300df49298418feac, took 0ms) 2014-12-23T16:53:59.993+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e2300df49298418feac 2014-12-23T16:53:59.993+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:53:59.996+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:53:59.998+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:00.002+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:00.002+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:00.003+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:00.003+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e2300df49298418feac 2014-12-23T16:54:00.052+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:00.052+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e2300df49298418feac through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-17' 2014-12-23T16:54:00.356+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:00-54999e289a8f659f34bfb2e7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353640356), what: "moveChunk.commit", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:00.428+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:00.428+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:00.428+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-17 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 535 2014-12-23T16:54:00.428+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-17 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:00.428+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:00.698+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:00.698+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:00-54999e289a8f659f34bfb2e8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353640698), what: "moveChunk.from", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 848, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 425, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:00.755+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2300df49298418feac') } ntoreturn:1 keyUpdates:0 reslen:37 1611ms 2014-12-23T16:54:00.757+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:00.757+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2300df49298418feac') } 2014-12-23T16:54:01.363+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e289a8f659f34bfb2e9 2014-12-23T16:54:01.363+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:01-54999e299a8f659f34bfb2ea", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353641363), what: "moveChunk.start", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:01.420+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 2|0||54999e2300df49298418feac, current metadata version is 2|0||54999e2300df49298418feac 2014-12-23T16:54:01.421+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-17 from shard version 2|0||54999e2300df49298418feac to shard version 2|1||54999e2300df49298418feac 2014-12-23T16:54:01.421+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e2300df49298418feac, took 0ms 2014-12-23T16:54:01.421+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e2300df49298418feac 2014-12-23T16:54:01.421+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:01.423+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:01.426+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:01.430+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:01.438+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:01.455+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:01.455+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:01.455+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:01.455+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e2300df49298418feac 2014-12-23T16:54:01.505+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:01.505+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e2300df49298418feac through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-17' 2014-12-23T16:54:01.810+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:01-54999e299a8f659f34bfb2eb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353641810), what: "moveChunk.commit", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:01.861+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:01.861+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:01.862+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-17 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 543 2014-12-23T16:54:01.862+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-17 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:01.862+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:02.165+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:02.165+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:02-54999e2a9a8f659f34bfb2ec", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353642165), what: "moveChunk.from", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 406, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:02.219+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2300df49298418feac') } ntoreturn:1 keyUpdates:0 reslen:37 1461ms 2014-12-23T16:54:02.220+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-17", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-17-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e2300df49298418feac') } 2014-12-23T16:54:02.781+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e2a9a8f659f34bfb2ed 2014-12-23T16:54:02.781+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 3|0||54999e2300df49298418feac, current metadata version is 3|0||54999e2300df49298418feac 2014-12-23T16:54:02.782+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-17 from shard version 3|0||54999e2300df49298418feac to shard version 3|1||54999e2300df49298418feac 2014-12-23T16:54:02.782+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e2300df49298418feac, took 0ms 2014-12-23T16:54:02.782+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e2300df49298418feac 2014-12-23T16:54:02.881+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:54:02 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:54:03.087+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:03-54999e2b9a8f659f34bfb2ee", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353643087), what: "split", ns: "benchdb1.COL-17", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e2300df49298418feac') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e2300df49298418feac') } } } 2014-12-23T16:54:03.475+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:03.475+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-17", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-17-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e2300df49298418feac') } ntoreturn:1 keyUpdates:0 reslen:37 1255ms 2014-12-23T16:54:06.039+0000 I INDEX [conn2] build index on: benchdb1.COL-18 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-18" } 2014-12-23T16:54:06.039+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:06.040+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:06.338+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-18 with requested shard version 1|2||54999e2b00df49298418feb0, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:06.339+0000 I SHARDING [conn2] collection benchdb1.COL-18 was previously unsharded, new metadata loaded with shard version 1|2||54999e2b00df49298418feb0 2014-12-23T16:54:06.339+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e2b00df49298418feb0, took 0ms 2014-12-23T16:54:06.393+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:06.393+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2b00df49298418feb0') } 2014-12-23T16:54:07.302+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e2e9a8f659f34bfb2f0 2014-12-23T16:54:07.302+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:07-54999e2f9a8f659f34bfb2f1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353647302), what: "moveChunk.start", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:07.359+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 1|2||54999e2b00df49298418feb0, current metadata version is 1|2||54999e2b00df49298418feb0 2014-12-23T16:54:07.359+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-18 already up to date (shard version : 1|2||54999e2b00df49298418feb0, took 0ms) 2014-12-23T16:54:07.360+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e2b00df49298418feb0 2014-12-23T16:54:07.360+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:07.363+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:07.365+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:07.369+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:07.369+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:07.370+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:07.370+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e2b00df49298418feb0 2014-12-23T16:54:07.421+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:07.421+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e2b00df49298418feb0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-18' 2014-12-23T16:54:07.725+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:07-54999e2f9a8f659f34bfb2f2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353647725), what: "moveChunk.commit", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:07.782+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:07.782+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:07.783+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-18 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 562 2014-12-23T16:54:07.783+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-18 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:07.783+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:08.086+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:08.086+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:08-54999e309a8f659f34bfb2f3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353648086), what: "moveChunk.from", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 966, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 412, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:08.135+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2b00df49298418feb0') } ntoreturn:1 keyUpdates:0 reslen:37 1742ms 2014-12-23T16:54:08.136+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:08.137+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2b00df49298418feb0') } 2014-12-23T16:54:08.744+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e309a8f659f34bfb2f4 2014-12-23T16:54:08.744+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:08-54999e309a8f659f34bfb2f5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353648744), what: "moveChunk.start", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:08.797+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 2|0||54999e2b00df49298418feb0, current metadata version is 2|0||54999e2b00df49298418feb0 2014-12-23T16:54:08.798+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-18 from shard version 2|0||54999e2b00df49298418feb0 to shard version 2|1||54999e2b00df49298418feb0 2014-12-23T16:54:08.798+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e2b00df49298418feb0, took 0ms 2014-12-23T16:54:08.798+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e2b00df49298418feb0 2014-12-23T16:54:08.798+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:08.800+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:08.802+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:08.807+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:08.815+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:08.832+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:08.832+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:08.832+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:08.832+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e2b00df49298418feb0 2014-12-23T16:54:08.880+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:08.880+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e2b00df49298418feb0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-18' 2014-12-23T16:54:09.186+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:09-54999e319a8f659f34bfb2f6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353649186), what: "moveChunk.commit", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:09.249+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:09.249+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:09.249+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-18 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 570 2014-12-23T16:54:09.249+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-18 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:09.249+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:09.505+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:09.505+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:09-54999e319a8f659f34bfb2f7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353649505), what: "moveChunk.from", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 660, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 417, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:09.561+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e2b00df49298418feb0') } ntoreturn:1 keyUpdates:0 reslen:37 1424ms 2014-12-23T16:54:09.563+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-18", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-18-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e2b00df49298418feb0') } 2014-12-23T16:54:10.029+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e319a8f659f34bfb2f8 2014-12-23T16:54:10.029+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 3|0||54999e2b00df49298418feb0, current metadata version is 3|0||54999e2b00df49298418feb0 2014-12-23T16:54:10.030+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-18 from shard version 3|0||54999e2b00df49298418feb0 to shard version 3|1||54999e2b00df49298418feb0 2014-12-23T16:54:10.030+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e2b00df49298418feb0, took 0ms 2014-12-23T16:54:10.030+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e2b00df49298418feb0 2014-12-23T16:54:10.199+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:10-54999e329a8f659f34bfb2f9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353650199), what: "split", ns: "benchdb1.COL-18", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e2b00df49298418feb0') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e2b00df49298418feb0') } } } 2014-12-23T16:54:10.423+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:10.423+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-18", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-18-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e2b00df49298418feb0') } ntoreturn:1 keyUpdates:0 reslen:37 860ms 2014-12-23T16:54:12.833+0000 I INDEX [conn2] build index on: benchdb1.COL-19 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-19" } 2014-12-23T16:54:12.833+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:12.834+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:13.135+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-19 with requested shard version 1|2||54999e3100df49298418feb5, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:13.136+0000 I SHARDING [conn2] collection benchdb1.COL-19 was previously unsharded, new metadata loaded with shard version 1|2||54999e3100df49298418feb5 2014-12-23T16:54:13.136+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e3100df49298418feb5, took 0ms 2014-12-23T16:54:13.182+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:13.182+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3100df49298418feb5') } 2014-12-23T16:54:14.092+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e359a8f659f34bfb2fb 2014-12-23T16:54:14.092+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:14-54999e369a8f659f34bfb2fc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353654092), what: "moveChunk.start", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:14.146+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 1|2||54999e3100df49298418feb5, current metadata version is 1|2||54999e3100df49298418feb5 2014-12-23T16:54:14.147+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-19 already up to date (shard version : 1|2||54999e3100df49298418feb5, took 0ms) 2014-12-23T16:54:14.147+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e3100df49298418feb5 2014-12-23T16:54:14.147+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:14.150+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:14.152+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:14.157+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:14.157+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:14.157+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:14.157+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e3100df49298418feb5 2014-12-23T16:54:14.210+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:14.210+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e3100df49298418feb5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-19' 2014-12-23T16:54:14.515+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:14-54999e369a8f659f34bfb2fd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353654515), what: "moveChunk.commit", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:14.572+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:14.572+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:14.572+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-19 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 591 2014-12-23T16:54:14.572+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-19 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:14.572+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:14.876+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:14.876+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:14-54999e369a8f659f34bfb2fe", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353654876), what: "moveChunk.from", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 964, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 415, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:14.930+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3100df49298418feb5') } ntoreturn:1 keyUpdates:0 reslen:37 1748ms 2014-12-23T16:54:14.932+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:14.932+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3100df49298418feb5') } 2014-12-23T16:54:15.539+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e369a8f659f34bfb2ff 2014-12-23T16:54:15.539+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:15-54999e379a8f659f34bfb300", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353655539), what: "moveChunk.start", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:15.594+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 2|0||54999e3100df49298418feb5, current metadata version is 2|0||54999e3100df49298418feb5 2014-12-23T16:54:15.595+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-19 from shard version 2|0||54999e3100df49298418feb5 to shard version 2|1||54999e3100df49298418feb5 2014-12-23T16:54:15.595+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e3100df49298418feb5, took 0ms 2014-12-23T16:54:15.595+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e3100df49298418feb5 2014-12-23T16:54:15.595+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:15.598+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:15.600+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:15.604+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:15.613+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:15.629+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:15.629+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:15.630+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:15.630+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e3100df49298418feb5 2014-12-23T16:54:15.689+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:15.689+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e3100df49298418feb5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-19' 2014-12-23T16:54:15.993+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:15-54999e379a8f659f34bfb301", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353655993), what: "moveChunk.commit", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:16.054+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:16.054+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:16.054+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-19 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 599 2014-12-23T16:54:16.054+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-19 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:16.054+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:16.357+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:16.358+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:16-54999e389a8f659f34bfb302", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353656358), what: "moveChunk.from", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:16.408+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3100df49298418feb5') } ntoreturn:1 keyUpdates:0 reslen:37 1476ms 2014-12-23T16:54:16.410+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-19", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-19-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e3100df49298418feb5') } 2014-12-23T16:54:17.018+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e389a8f659f34bfb303 2014-12-23T16:54:17.018+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 3|0||54999e3100df49298418feb5, current metadata version is 3|0||54999e3100df49298418feb5 2014-12-23T16:54:17.019+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-19 from shard version 3|0||54999e3100df49298418feb5 to shard version 3|1||54999e3100df49298418feb5 2014-12-23T16:54:17.019+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e3100df49298418feb5, took 0ms 2014-12-23T16:54:17.019+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e3100df49298418feb5 2014-12-23T16:54:17.324+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:17-54999e399a8f659f34bfb304", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353657324), what: "split", ns: "benchdb1.COL-19", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e3100df49298418feb5') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e3100df49298418feb5') } } } 2014-12-23T16:54:17.709+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:17.709+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-19", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-19-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e3100df49298418feb5') } ntoreturn:1 keyUpdates:0 reslen:37 1298ms 2014-12-23T16:54:20.311+0000 I INDEX [conn2] build index on: benchdb1.COL-20 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-20" } 2014-12-23T16:54:20.311+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:20.311+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:20.611+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-20 with requested shard version 1|2||54999e3900df49298418feb8, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:20.612+0000 I SHARDING [conn2] collection benchdb1.COL-20 was previously unsharded, new metadata loaded with shard version 1|2||54999e3900df49298418feb8 2014-12-23T16:54:20.612+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e3900df49298418feb8, took 0ms 2014-12-23T16:54:20.655+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:20.655+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3900df49298418feb8') } 2014-12-23T16:54:21.401+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e3c9a8f659f34bfb306 2014-12-23T16:54:21.401+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:21-54999e3d9a8f659f34bfb307", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353661401), what: "moveChunk.start", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:21.463+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 1|2||54999e3900df49298418feb8, current metadata version is 1|2||54999e3900df49298418feb8 2014-12-23T16:54:21.464+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-20 already up to date (shard version : 1|2||54999e3900df49298418feb8, took 0ms) 2014-12-23T16:54:21.464+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e3900df49298418feb8 2014-12-23T16:54:21.465+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:21.474+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:21.477+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:21.481+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:21.490+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:21.490+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:21.490+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:21.490+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e3900df49298418feb8 2014-12-23T16:54:21.576+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:21.576+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e3900df49298418feb8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-20' 2014-12-23T16:54:21.770+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:21-54999e3d9a8f659f34bfb308", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353661770), what: "moveChunk.commit", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:21.824+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:21.824+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:21.824+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-20 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 618 2014-12-23T16:54:21.824+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-20 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:21.824+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:21.978+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:21.978+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:21-54999e3d9a8f659f34bfb309", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353661978), what: "moveChunk.from", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 809, step 3 of 6: 8, step 4 of 6: 16, step 5 of 6: 333, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:22.029+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3900df49298418feb8') } ntoreturn:1 keyUpdates:0 reslen:37 1373ms 2014-12-23T16:54:22.030+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:22.030+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3900df49298418feb8') } 2014-12-23T16:54:22.326+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('54999e3900df49298418feb8') } 2014-12-23T16:54:22.327+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-20 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-20 is taken. 2014-12-23T16:54:22.327+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:22-54999e3e9a8f659f34bfb30b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353662327), what: "moveChunk.from", ns: "benchdb1.COL-20", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-20 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKe..." } } 2014-12-23T16:54:22.358+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e3e9a8f659f34bfb30a 2014-12-23T16:54:22.358+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:22-54999e3e9a8f659f34bfb30c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353662358), what: "moveChunk.start", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:22.441+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 2|0||54999e3900df49298418feb8, current metadata version is 2|0||54999e3900df49298418feb8 2014-12-23T16:54:22.442+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-20 from shard version 2|0||54999e3900df49298418feb8 to shard version 2|1||54999e3900df49298418feb8 2014-12-23T16:54:22.442+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e3900df49298418feb8, took 0ms 2014-12-23T16:54:22.442+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e3900df49298418feb8 2014-12-23T16:54:22.442+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:22.445+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:22.447+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:22.452+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:22.460+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:22.476+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:22.476+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:22.477+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:22.477+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e3900df49298418feb8 2014-12-23T16:54:22.527+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:22.527+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e3900df49298418feb8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-20' 2014-12-23T16:54:22.831+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:22-54999e3e9a8f659f34bfb30d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353662831), what: "moveChunk.commit", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:22.901+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:22.901+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:22.901+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-20 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 628 2014-12-23T16:54:22.901+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-20 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:22.901+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:23.204+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:23.204+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:23-54999e3f9a8f659f34bfb30e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353663204), what: "moveChunk.from", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 411, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:23.257+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e3900df49298418feb8') } ntoreturn:1 keyUpdates:0 reslen:37 1227ms 2014-12-23T16:54:23.259+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-20", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-20-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e3900df49298418feb8') } 2014-12-23T16:54:23.831+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e3f9a8f659f34bfb30f 2014-12-23T16:54:23.832+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 3|0||54999e3900df49298418feb8, current metadata version is 3|0||54999e3900df49298418feb8 2014-12-23T16:54:23.832+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-20 from shard version 3|0||54999e3900df49298418feb8 to shard version 3|1||54999e3900df49298418feb8 2014-12-23T16:54:23.832+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e3900df49298418feb8, took 0ms 2014-12-23T16:54:23.832+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e3900df49298418feb8 2014-12-23T16:54:24.136+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:24-54999e409a8f659f34bfb310", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353664136), what: "split", ns: "benchdb1.COL-20", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e3900df49298418feb8') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e3900df49298418feb8') } } } 2014-12-23T16:54:24.505+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:24.505+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-20", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-20-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e3900df49298418feb8') } ntoreturn:1 keyUpdates:0 reslen:37 1245ms 2014-12-23T16:54:27.098+0000 I INDEX [conn2] build index on: benchdb1.COL-21 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-21" } 2014-12-23T16:54:27.098+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:27.099+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:27.393+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-21 with requested shard version 1|2||54999e4000df49298418febd, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:27.394+0000 I SHARDING [conn2] collection benchdb1.COL-21 was previously unsharded, new metadata loaded with shard version 1|2||54999e4000df49298418febd 2014-12-23T16:54:27.394+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e4000df49298418febd, took 0ms 2014-12-23T16:54:27.442+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:27.442+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4000df49298418febd') } 2014-12-23T16:54:28.352+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e439a8f659f34bfb312 2014-12-23T16:54:28.352+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:28-54999e449a8f659f34bfb313", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353668352), what: "moveChunk.start", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:28.418+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 1|2||54999e4000df49298418febd, current metadata version is 1|2||54999e4000df49298418febd 2014-12-23T16:54:28.419+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-21 already up to date (shard version : 1|2||54999e4000df49298418febd, took 0ms) 2014-12-23T16:54:28.419+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e4000df49298418febd 2014-12-23T16:54:28.419+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:28.422+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:28.424+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:28.428+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:28.428+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:28.429+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:28.429+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e4000df49298418febd 2014-12-23T16:54:28.479+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:28.479+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e4000df49298418febd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-21' 2014-12-23T16:54:28.783+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:28-54999e449a8f659f34bfb314", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353668783), what: "moveChunk.commit", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:28.839+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:28.839+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:28.839+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-21 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 648 2014-12-23T16:54:28.839+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-21 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:28.839+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:29.142+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:29.142+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:29-54999e459a8f659f34bfb315", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353669142), what: "moveChunk.from", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 976, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 410, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:29.187+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4000df49298418febd') } ntoreturn:1 keyUpdates:0 reslen:37 1745ms 2014-12-23T16:54:29.189+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:29.189+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4000df49298418febd') } 2014-12-23T16:54:29.795+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e459a8f659f34bfb316 2014-12-23T16:54:29.795+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:29-54999e459a8f659f34bfb317", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353669795), what: "moveChunk.start", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:29.854+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 2|0||54999e4000df49298418febd, current metadata version is 2|0||54999e4000df49298418febd 2014-12-23T16:54:29.855+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-21 from shard version 2|0||54999e4000df49298418febd to shard version 2|1||54999e4000df49298418febd 2014-12-23T16:54:29.855+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e4000df49298418febd, took 0ms 2014-12-23T16:54:29.855+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e4000df49298418febd 2014-12-23T16:54:29.855+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:29.857+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:29.860+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:29.864+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:29.872+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:29.889+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:29.889+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:29.889+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:29.889+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e4000df49298418febd 2014-12-23T16:54:29.949+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:29.949+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e4000df49298418febd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-21' 2014-12-23T16:54:30.253+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:30-54999e469a8f659f34bfb318", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353670253), what: "moveChunk.commit", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:30.311+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:30.311+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:30.311+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-21 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 656 2014-12-23T16:54:30.311+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-21 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:30.311+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:30.614+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:30.614+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:30-54999e469a8f659f34bfb319", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353670614), what: "moveChunk.from", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 665, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 422, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:30.666+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4000df49298418febd') } ntoreturn:1 keyUpdates:0 reslen:37 1477ms 2014-12-23T16:54:30.668+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-21", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-21-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e4000df49298418febd') } 2014-12-23T16:54:31.274+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e469a8f659f34bfb31a 2014-12-23T16:54:31.274+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 3|0||54999e4000df49298418febd, current metadata version is 3|0||54999e4000df49298418febd 2014-12-23T16:54:31.275+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-21 from shard version 3|0||54999e4000df49298418febd to shard version 3|1||54999e4000df49298418febd 2014-12-23T16:54:31.275+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e4000df49298418febd, took 0ms 2014-12-23T16:54:31.275+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e4000df49298418febd 2014-12-23T16:54:31.579+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:31-54999e479a8f659f34bfb31b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353671579), what: "split", ns: "benchdb1.COL-21", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e4000df49298418febd') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e4000df49298418febd') } } } 2014-12-23T16:54:31.961+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:31.961+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-21", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-21-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e4000df49298418febd') } ntoreturn:1 keyUpdates:0 reslen:37 1293ms 2014-12-23T16:54:33.308+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:54:32 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:54:33.899+0000 I INDEX [conn2] build index on: benchdb1.COL-22 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-22" } 2014-12-23T16:54:33.899+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:33.899+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:34.187+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-22 with requested shard version 1|2||54999e4600df49298418fec1, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:34.187+0000 I SHARDING [conn2] collection benchdb1.COL-22 was previously unsharded, new metadata loaded with shard version 1|2||54999e4600df49298418fec1 2014-12-23T16:54:34.187+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e4600df49298418fec1, took 0ms 2014-12-23T16:54:34.230+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:34.230+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4600df49298418fec1') } 2014-12-23T16:54:34.852+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e4a9a8f659f34bfb31d 2014-12-23T16:54:34.852+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:34-54999e4a9a8f659f34bfb31e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353674852), what: "moveChunk.start", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:34.924+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 1|2||54999e4600df49298418fec1, current metadata version is 1|2||54999e4600df49298418fec1 2014-12-23T16:54:34.925+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-22 already up to date (shard version : 1|2||54999e4600df49298418fec1, took 0ms) 2014-12-23T16:54:34.925+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e4600df49298418fec1 2014-12-23T16:54:34.926+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:34.928+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:34.931+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:34.935+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:34.935+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:34.935+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:34.935+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e4600df49298418fec1 2014-12-23T16:54:34.987+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:34.987+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e4600df49298418fec1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-22' 2014-12-23T16:54:35.291+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:35-54999e4b9a8f659f34bfb31f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353675291), what: "moveChunk.commit", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:35.349+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:35.349+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:35.349+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-22 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 677 2014-12-23T16:54:35.349+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-22 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:35.349+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:35.652+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:35.652+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:35-54999e4b9a8f659f34bfb320", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353675652), what: "moveChunk.from", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 695, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 413, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:35.703+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4600df49298418fec1') } ntoreturn:1 keyUpdates:0 reslen:37 1472ms 2014-12-23T16:54:35.704+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:35.704+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4600df49298418fec1') } 2014-12-23T16:54:36.311+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e4b9a8f659f34bfb321 2014-12-23T16:54:36.311+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:36-54999e4c9a8f659f34bfb322", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353676311), what: "moveChunk.start", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:36.363+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 2|0||54999e4600df49298418fec1, current metadata version is 2|0||54999e4600df49298418fec1 2014-12-23T16:54:36.363+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-22 from shard version 2|0||54999e4600df49298418fec1 to shard version 2|1||54999e4600df49298418fec1 2014-12-23T16:54:36.363+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e4600df49298418fec1, took 0ms 2014-12-23T16:54:36.363+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e4600df49298418fec1 2014-12-23T16:54:36.363+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:36.366+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:36.368+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:36.372+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:36.381+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:36.397+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:36.397+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:36.398+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:36.398+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e4600df49298418fec1 2014-12-23T16:54:36.446+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:36.446+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e4600df49298418fec1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-22' 2014-12-23T16:54:36.750+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:36-54999e4c9a8f659f34bfb323", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353676750), what: "moveChunk.commit", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:36.807+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:36.807+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:36.807+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-22 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 685 2014-12-23T16:54:36.808+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-22 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:36.808+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:37.110+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:37.110+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:37-54999e4d9a8f659f34bfb324", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353677110), what: "moveChunk.from", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 658, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 410, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:37.188+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4600df49298418fec1') } ntoreturn:1 keyUpdates:0 reslen:37 1483ms 2014-12-23T16:54:37.190+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-22", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-22-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e4600df49298418fec1') } 2014-12-23T16:54:37.796+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e4d9a8f659f34bfb325 2014-12-23T16:54:37.796+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 3|0||54999e4600df49298418fec1, current metadata version is 3|0||54999e4600df49298418fec1 2014-12-23T16:54:37.797+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-22 from shard version 3|0||54999e4600df49298418fec1 to shard version 3|1||54999e4600df49298418fec1 2014-12-23T16:54:37.797+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e4600df49298418fec1, took 0ms 2014-12-23T16:54:37.797+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e4600df49298418fec1 2014-12-23T16:54:38.101+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:38-54999e4e9a8f659f34bfb326", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353678101), what: "split", ns: "benchdb1.COL-22", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e4600df49298418fec1') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e4600df49298418fec1') } } } 2014-12-23T16:54:38.478+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:38.478+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-22", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-22-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e4600df49298418fec1') } ntoreturn:1 keyUpdates:0 reslen:37 1288ms 2014-12-23T16:54:41.098+0000 I INDEX [conn2] build index on: benchdb1.COL-23 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-23" } 2014-12-23T16:54:41.098+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:41.099+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:41.401+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-23 with requested shard version 1|2||54999e4e00df49298418fec5, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:41.401+0000 I SHARDING [conn2] collection benchdb1.COL-23 was previously unsharded, new metadata loaded with shard version 1|2||54999e4e00df49298418fec5 2014-12-23T16:54:41.401+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e4e00df49298418fec5, took 0ms 2014-12-23T16:54:41.444+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:41.445+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4e00df49298418fec5') } 2014-12-23T16:54:42.355+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e519a8f659f34bfb328 2014-12-23T16:54:42.355+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:42-54999e529a8f659f34bfb329", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353682355), what: "moveChunk.start", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:42.420+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 1|2||54999e4e00df49298418fec5, current metadata version is 1|2||54999e4e00df49298418fec5 2014-12-23T16:54:42.421+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-23 already up to date (shard version : 1|2||54999e4e00df49298418fec5, took 0ms) 2014-12-23T16:54:42.421+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e4e00df49298418fec5 2014-12-23T16:54:42.421+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:42.424+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:42.426+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:42.431+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:42.431+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:42.431+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:42.431+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e4e00df49298418fec5 2014-12-23T16:54:42.482+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:42.482+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e4e00df49298418fec5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-23' 2014-12-23T16:54:42.787+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:42-54999e529a8f659f34bfb32a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353682787), what: "moveChunk.commit", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:42.843+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:42.843+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:42.843+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-23 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 704 2014-12-23T16:54:42.843+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-23 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:42.843+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:43.146+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:43.146+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:43-54999e539a8f659f34bfb32b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353683146), what: "moveChunk.from", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 976, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 412, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:43.193+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4e00df49298418fec5') } ntoreturn:1 keyUpdates:0 reslen:37 1748ms 2014-12-23T16:54:43.194+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:43.195+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4e00df49298418fec5') } 2014-12-23T16:54:43.801+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e539a8f659f34bfb32c 2014-12-23T16:54:43.801+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:43-54999e539a8f659f34bfb32d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353683801), what: "moveChunk.start", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:43.854+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 2|0||54999e4e00df49298418fec5, current metadata version is 2|0||54999e4e00df49298418fec5 2014-12-23T16:54:43.855+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-23 from shard version 2|0||54999e4e00df49298418fec5 to shard version 2|1||54999e4e00df49298418fec5 2014-12-23T16:54:43.855+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e4e00df49298418fec5, took 0ms 2014-12-23T16:54:43.855+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e4e00df49298418fec5 2014-12-23T16:54:43.855+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:43.857+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:43.859+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:43.864+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:43.872+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:43.889+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:43.889+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:43.889+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:43.889+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e4e00df49298418fec5 2014-12-23T16:54:43.944+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:43.944+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e4e00df49298418fec5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-23' 2014-12-23T16:54:44.248+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:44-54999e549a8f659f34bfb32e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353684248), what: "moveChunk.commit", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:44.304+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:44.304+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:44.304+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-23 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 712 2014-12-23T16:54:44.304+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-23 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:44.304+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:44.609+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:44.609+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:44-54999e549a8f659f34bfb32f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353684609), what: "moveChunk.from", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 660, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 415, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:44.656+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e4e00df49298418fec5') } ntoreturn:1 keyUpdates:0 reslen:37 1462ms 2014-12-23T16:54:44.658+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-23", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-23-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e4e00df49298418fec5') } 2014-12-23T16:54:45.214+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e549a8f659f34bfb330 2014-12-23T16:54:45.214+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 3|0||54999e4e00df49298418fec5, current metadata version is 3|0||54999e4e00df49298418fec5 2014-12-23T16:54:45.215+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-23 from shard version 3|0||54999e4e00df49298418fec5 to shard version 3|1||54999e4e00df49298418fec5 2014-12-23T16:54:45.215+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e4e00df49298418fec5, took 0ms 2014-12-23T16:54:45.215+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e4e00df49298418fec5 2014-12-23T16:54:45.442+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:45-54999e559a8f659f34bfb331", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353685442), what: "split", ns: "benchdb1.COL-23", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e4e00df49298418fec5') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e4e00df49298418fec5') } } } 2014-12-23T16:54:45.646+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:45.646+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-23", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-23-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e4e00df49298418fec5') } ntoreturn:1 keyUpdates:0 reslen:37 988ms 2014-12-23T16:54:47.577+0000 I INDEX [conn2] build index on: benchdb1.COL-24 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-24" } 2014-12-23T16:54:47.577+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:47.578+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:47.873+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-24 with requested shard version 1|2||54999e5400df49298418feca, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:47.874+0000 I SHARDING [conn2] collection benchdb1.COL-24 was previously unsharded, new metadata loaded with shard version 1|2||54999e5400df49298418feca 2014-12-23T16:54:47.874+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e5400df49298418feca, took 0ms 2014-12-23T16:54:47.915+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:47.916+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5400df49298418feca') } 2014-12-23T16:54:48.825+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e589a8f659f34bfb333 2014-12-23T16:54:48.825+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:48-54999e589a8f659f34bfb334", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353688825), what: "moveChunk.start", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:48.877+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 1|2||54999e5400df49298418feca, current metadata version is 1|2||54999e5400df49298418feca 2014-12-23T16:54:48.878+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-24 already up to date (shard version : 1|2||54999e5400df49298418feca, took 0ms) 2014-12-23T16:54:48.878+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e5400df49298418feca 2014-12-23T16:54:48.879+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:48.881+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:48.884+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:48.888+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:48.888+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:48.888+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:48.888+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e5400df49298418feca 2014-12-23T16:54:48.939+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:48.939+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e5400df49298418feca through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-24' 2014-12-23T16:54:49.243+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:49-54999e599a8f659f34bfb335", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353689243), what: "moveChunk.commit", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:49.297+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:49.297+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:49.297+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-24 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 733 2014-12-23T16:54:49.297+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-24 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:49.297+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:49.600+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:49.600+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:49-54999e599a8f659f34bfb336", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353689600), what: "moveChunk.from", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 962, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 409, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:49.654+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5400df49298418feca') } ntoreturn:1 keyUpdates:0 reslen:37 1738ms 2014-12-23T16:54:49.655+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:49.656+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5400df49298418feca') } 2014-12-23T16:54:50.244+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e599a8f659f34bfb337 2014-12-23T16:54:50.244+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:50-54999e5a9a8f659f34bfb338", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353690244), what: "moveChunk.start", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:50.296+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 2|0||54999e5400df49298418feca, current metadata version is 2|0||54999e5400df49298418feca 2014-12-23T16:54:50.297+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-24 from shard version 2|0||54999e5400df49298418feca to shard version 2|1||54999e5400df49298418feca 2014-12-23T16:54:50.297+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e5400df49298418feca, took 0ms 2014-12-23T16:54:50.297+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e5400df49298418feca 2014-12-23T16:54:50.297+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:50.299+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:50.302+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:50.306+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:50.314+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:50.331+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:50.331+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:50.331+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:50.331+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e5400df49298418feca 2014-12-23T16:54:50.381+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:50.382+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e5400df49298418feca through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-24' 2014-12-23T16:54:50.686+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:50-54999e5a9a8f659f34bfb339", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353690686), what: "moveChunk.commit", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:50.744+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:50.744+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:50.744+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-24 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 741 2014-12-23T16:54:50.744+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-24 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:50.744+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:51.047+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:51.047+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:51-54999e5b9a8f659f34bfb33a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353691047), what: "moveChunk.from", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 641, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 413, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:51.095+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5400df49298418feca') } ntoreturn:1 keyUpdates:0 reslen:37 1440ms 2014-12-23T16:54:51.097+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-24", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-24-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e5400df49298418feca') } 2014-12-23T16:54:51.704+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e5b9a8f659f34bfb33b 2014-12-23T16:54:51.704+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 3|0||54999e5400df49298418feca, current metadata version is 3|0||54999e5400df49298418feca 2014-12-23T16:54:51.704+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-24 from shard version 3|0||54999e5400df49298418feca to shard version 3|1||54999e5400df49298418feca 2014-12-23T16:54:51.705+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e5400df49298418feca, took 0ms 2014-12-23T16:54:51.705+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e5400df49298418feca 2014-12-23T16:54:52.009+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:52-54999e5c9a8f659f34bfb33c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353692009), what: "split", ns: "benchdb1.COL-24", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e5400df49298418feca') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e5400df49298418feca') } } } 2014-12-23T16:54:52.394+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:52.394+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-24", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-24-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e5400df49298418feca') } ntoreturn:1 keyUpdates:0 reslen:37 1297ms 2014-12-23T16:54:55.032+0000 I INDEX [conn2] build index on: benchdb1.COL-25 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-25" } 2014-12-23T16:54:55.032+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:54:55.033+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:54:55.327+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-25 with requested shard version 1|2||54999e5c00df49298418fecd, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:54:55.328+0000 I SHARDING [conn2] collection benchdb1.COL-25 was previously unsharded, new metadata loaded with shard version 1|2||54999e5c00df49298418fecd 2014-12-23T16:54:55.328+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e5c00df49298418fecd, took 0ms 2014-12-23T16:54:55.366+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:55.367+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5c00df49298418fecd') } 2014-12-23T16:54:56.276+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e5f9a8f659f34bfb33e 2014-12-23T16:54:56.276+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:56-54999e609a8f659f34bfb33f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353696276), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:54:56.341+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 1|2||54999e5c00df49298418fecd, current metadata version is 1|2||54999e5c00df49298418fecd 2014-12-23T16:54:56.341+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-25 already up to date (shard version : 1|2||54999e5c00df49298418fecd, took 0ms) 2014-12-23T16:54:56.341+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e5c00df49298418fecd 2014-12-23T16:54:56.342+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:56.344+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:56.347+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:56.351+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:56.351+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:56.351+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:56.351+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e5c00df49298418fecd 2014-12-23T16:54:56.402+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:56.402+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e5c00df49298418fecd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-25' 2014-12-23T16:54:56.623+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:56-54999e609a8f659f34bfb340", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353696623), what: "moveChunk.commit", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:56.690+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:56.691+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:56.691+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-25 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 760 2014-12-23T16:54:56.691+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-25 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:54:56.691+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:56.944+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:56.944+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:56-54999e609a8f659f34bfb341", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353696944), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 974, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 339, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:54:57.036+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5c00df49298418fecd') } ntoreturn:1 keyUpdates:0 reslen:37 1669ms 2014-12-23T16:54:57.037+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:54:57.038+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5c00df49298418fecd') } 2014-12-23T16:54:57.362+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e619a8f659f34bfb342 2014-12-23T16:54:57.362+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:57-54999e619a8f659f34bfb343", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353697362), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:54:57.413+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 2|0||54999e5c00df49298418fecd, current metadata version is 2|0||54999e5c00df49298418fecd 2014-12-23T16:54:57.414+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-25 from shard version 2|0||54999e5c00df49298418fecd to shard version 2|1||54999e5c00df49298418fecd 2014-12-23T16:54:57.414+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e5c00df49298418fecd, took 0ms 2014-12-23T16:54:57.414+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e5c00df49298418fecd 2014-12-23T16:54:57.414+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:54:57.416+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:57.419+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:57.423+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:57.431+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:57.448+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:54:57.448+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:54:57.448+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:54:57.448+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e5c00df49298418fecd 2014-12-23T16:54:57.509+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:54:57.509+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e5c00df49298418fecd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-25' 2014-12-23T16:54:57.681+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:57-54999e619a8f659f34bfb344", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353697681), what: "moveChunk.commit", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:54:57.733+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:57.733+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:54:57.733+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-25 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 769 2014-12-23T16:54:57.733+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-25 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:54:57.733+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:54:57.884+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:57.884+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:57-54999e619a8f659f34bfb345", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353697884), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 376, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 284, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:54:57.935+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e5c00df49298418fecd') } ntoreturn:1 keyUpdates:0 reslen:37 898ms 2014-12-23T16:54:57.937+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e5c00df49298418fecd') } 2014-12-23T16:54:58.331+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e619a8f659f34bfb346 2014-12-23T16:54:58.331+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 3|0||54999e5c00df49298418fecd, current metadata version is 3|0||54999e5c00df49298418fecd 2014-12-23T16:54:58.332+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-25 from shard version 3|0||54999e5c00df49298418fecd to shard version 3|1||54999e5c00df49298418fecd 2014-12-23T16:54:58.332+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e5c00df49298418fecd, took 0ms 2014-12-23T16:54:58.332+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e5c00df49298418fecd 2014-12-23T16:54:58.636+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:54:58-54999e629a8f659f34bfb347", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353698636), what: "split", ns: "benchdb1.COL-25", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e5c00df49298418fecd') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e5c00df49298418fecd') } } } 2014-12-23T16:54:59.029+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:54:59.029+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e5c00df49298418fecd') } ntoreturn:1 keyUpdates:0 reslen:37 1091ms 2014-12-23T16:55:01.494+0000 I INDEX [conn2] build index on: benchdb1.COL-26 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-26" } 2014-12-23T16:55:01.494+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:01.495+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:01.790+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-26 with requested shard version 1|2||54999e6200df49298418fed2, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:01.791+0000 I SHARDING [conn2] collection benchdb1.COL-26 was previously unsharded, new metadata loaded with shard version 1|2||54999e6200df49298418fed2 2014-12-23T16:55:01.791+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e6200df49298418fed2, took 0ms 2014-12-23T16:55:01.870+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:01.870+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6200df49298418fed2') } 2014-12-23T16:55:02.781+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e669a8f659f34bfb349 2014-12-23T16:55:02.781+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:02-54999e669a8f659f34bfb34a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353702781), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:02.839+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 1|2||54999e6200df49298418fed2, current metadata version is 1|2||54999e6200df49298418fed2 2014-12-23T16:55:02.840+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-26 already up to date (shard version : 1|2||54999e6200df49298418fed2, took 0ms) 2014-12-23T16:55:02.840+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e6200df49298418fed2 2014-12-23T16:55:02.841+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:02.844+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:02.847+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:02.851+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:02.851+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:02.852+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:02.852+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e6200df49298418fed2 2014-12-23T16:55:02.905+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:02.905+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e6200df49298418fed2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-26' 2014-12-23T16:55:03.211+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:03-54999e679a8f659f34bfb34b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353703211), what: "moveChunk.commit", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:03.266+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:03.267+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:03.267+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-26 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 788 2014-12-23T16:55:03.267+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-26 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:03.267+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:03.480+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:03.481+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:03-54999e679a8f659f34bfb34c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353703481), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 970, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 415, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:03.541+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6200df49298418fed2') } ntoreturn:1 keyUpdates:0 reslen:37 1670ms 2014-12-23T16:55:03.542+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:03.543+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6200df49298418fed2') } 2014-12-23T16:55:03.745+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:55:03 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:55:04.150+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e679a8f659f34bfb34d 2014-12-23T16:55:04.150+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:04-54999e689a8f659f34bfb34e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353704150), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:04.224+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 2|0||54999e6200df49298418fed2, current metadata version is 2|0||54999e6200df49298418fed2 2014-12-23T16:55:04.225+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-26 from shard version 2|0||54999e6200df49298418fed2 to shard version 2|1||54999e6200df49298418fed2 2014-12-23T16:55:04.225+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e6200df49298418fed2, took 0ms 2014-12-23T16:55:04.225+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e6200df49298418fed2 2014-12-23T16:55:04.225+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:04.228+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:04.230+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:04.235+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:04.243+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:04.259+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:04.259+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:04.260+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:04.260+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e6200df49298418fed2 2014-12-23T16:55:04.314+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:04.314+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e6200df49298418fed2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-26' 2014-12-23T16:55:04.619+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:04-54999e689a8f659f34bfb34f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353704619), what: "moveChunk.commit", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:04.680+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:04.680+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:04.680+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-26 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 796 2014-12-23T16:55:04.680+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-26 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:04.680+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:04.984+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:04.984+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:04-54999e689a8f659f34bfb350", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353704984), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 682, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 420, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:05.040+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6200df49298418fed2') } ntoreturn:1 keyUpdates:0 reslen:37 1498ms 2014-12-23T16:55:05.042+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e6200df49298418fed2') } 2014-12-23T16:55:05.550+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e699a8f659f34bfb351 2014-12-23T16:55:05.550+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 3|0||54999e6200df49298418fed2, current metadata version is 3|0||54999e6200df49298418fed2 2014-12-23T16:55:05.550+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-26 from shard version 3|0||54999e6200df49298418fed2 to shard version 3|1||54999e6200df49298418fed2 2014-12-23T16:55:05.550+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e6200df49298418fed2, took 0ms 2014-12-23T16:55:05.550+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e6200df49298418fed2 2014-12-23T16:55:05.855+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:05-54999e699a8f659f34bfb352", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353705855), what: "split", ns: "benchdb1.COL-26", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e6200df49298418fed2') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e6200df49298418fed2') } } } 2014-12-23T16:55:06.245+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:06.245+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e6200df49298418fed2') } ntoreturn:1 keyUpdates:0 reslen:37 1203ms 2014-12-23T16:55:08.832+0000 I INDEX [conn2] build index on: benchdb1.COL-27 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-27" } 2014-12-23T16:55:08.832+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:08.833+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:09.161+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-27 with requested shard version 1|2||54999e6900df49298418fed6, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:09.161+0000 I SHARDING [conn2] collection benchdb1.COL-27 was previously unsharded, new metadata loaded with shard version 1|2||54999e6900df49298418fed6 2014-12-23T16:55:09.161+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e6900df49298418fed6, took 0ms 2014-12-23T16:55:09.203+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:09.203+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6900df49298418fed6') } 2014-12-23T16:55:09.606+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e6d9a8f659f34bfb354 2014-12-23T16:55:09.607+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:09-54999e6d9a8f659f34bfb355", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353709607), what: "moveChunk.start", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:09.659+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 1|2||54999e6900df49298418fed6, current metadata version is 1|2||54999e6900df49298418fed6 2014-12-23T16:55:09.659+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-27 already up to date (shard version : 1|2||54999e6900df49298418fed6, took 0ms) 2014-12-23T16:55:09.659+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e6900df49298418fed6 2014-12-23T16:55:09.660+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:09.662+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:09.665+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:09.669+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:09.669+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:09.670+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:09.670+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e6900df49298418fed6 2014-12-23T16:55:09.713+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:09.713+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e6900df49298418fed6 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-27' 2014-12-23T16:55:09.876+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:09-54999e6d9a8f659f34bfb356", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353709876), what: "moveChunk.commit", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:09.919+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:09.920+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:09.920+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-27 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 817 2014-12-23T16:55:09.920+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-27 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:09.920+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:10.092+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:10.092+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:10-54999e6e9a8f659f34bfb357", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353710092), what: "moveChunk.from", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 456, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 250, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:10.146+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6900df49298418fed6') } ntoreturn:1 keyUpdates:0 reslen:37 943ms 2014-12-23T16:55:10.148+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:10.148+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6900df49298418fed6') } 2014-12-23T16:55:10.680+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e6e9a8f659f34bfb358 2014-12-23T16:55:10.680+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:10-54999e6e9a8f659f34bfb359", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353710680), what: "moveChunk.start", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:10.747+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 2|0||54999e6900df49298418fed6, current metadata version is 2|0||54999e6900df49298418fed6 2014-12-23T16:55:10.748+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-27 from shard version 2|0||54999e6900df49298418fed6 to shard version 2|1||54999e6900df49298418fed6 2014-12-23T16:55:10.748+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e6900df49298418fed6, took 0ms 2014-12-23T16:55:10.748+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e6900df49298418fed6 2014-12-23T16:55:10.748+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:10.750+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:10.752+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:10.757+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:10.765+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:10.782+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:10.782+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:10.782+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:10.782+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e6900df49298418fed6 2014-12-23T16:55:10.848+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:10.848+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e6900df49298418fed6 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-27' 2014-12-23T16:55:11.154+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:11-54999e6f9a8f659f34bfb35a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353711154), what: "moveChunk.commit", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:11.259+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:11.259+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:11.259+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-27 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 825 2014-12-23T16:55:11.259+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-27 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:11.259+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:11.562+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:11.562+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:11-54999e6f9a8f659f34bfb35b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353711562), what: "moveChunk.from", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 599, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 477, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:11.613+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e6900df49298418fed6') } ntoreturn:1 keyUpdates:0 reslen:37 1465ms 2014-12-23T16:55:11.615+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-27", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-27-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e6900df49298418fed6') } 2014-12-23T16:55:12.222+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e6f9a8f659f34bfb35c 2014-12-23T16:55:12.222+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 3|0||54999e6900df49298418fed6, current metadata version is 3|0||54999e6900df49298418fed6 2014-12-23T16:55:12.222+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-27 from shard version 3|0||54999e6900df49298418fed6 to shard version 3|1||54999e6900df49298418fed6 2014-12-23T16:55:12.222+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e6900df49298418fed6, took 0ms 2014-12-23T16:55:12.222+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e6900df49298418fed6 2014-12-23T16:55:12.527+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:12-54999e709a8f659f34bfb35d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353712527), what: "split", ns: "benchdb1.COL-27", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e6900df49298418fed6') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e6900df49298418fed6') } } } 2014-12-23T16:55:12.906+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:12.906+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-27", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-27-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e6900df49298418fed6') } ntoreturn:1 keyUpdates:0 reslen:37 1291ms 2014-12-23T16:55:15.496+0000 I INDEX [conn2] build index on: benchdb1.COL-28 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-28" } 2014-12-23T16:55:15.496+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:15.497+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:15.792+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-28 with requested shard version 1|2||54999e7000df49298418feda, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:15.793+0000 I SHARDING [conn2] collection benchdb1.COL-28 was previously unsharded, new metadata loaded with shard version 1|2||54999e7000df49298418feda 2014-12-23T16:55:15.793+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e7000df49298418feda, took 0ms 2014-12-23T16:55:15.836+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:15.836+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7000df49298418feda') } 2014-12-23T16:55:16.746+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e749a8f659f34bfb35f 2014-12-23T16:55:16.746+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:16-54999e749a8f659f34bfb360", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353716746), what: "moveChunk.start", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:16.801+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 1|2||54999e7000df49298418feda, current metadata version is 1|2||54999e7000df49298418feda 2014-12-23T16:55:16.802+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-28 already up to date (shard version : 1|2||54999e7000df49298418feda, took 0ms) 2014-12-23T16:55:16.802+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e7000df49298418feda 2014-12-23T16:55:16.803+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:16.805+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:16.808+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:16.812+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:16.812+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:16.812+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:16.812+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e7000df49298418feda 2014-12-23T16:55:16.867+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:16.867+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e7000df49298418feda through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-28' 2014-12-23T16:55:17.171+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:17-54999e759a8f659f34bfb361", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353717171), what: "moveChunk.commit", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:17.221+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:17.221+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:17.221+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-28 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 843 2014-12-23T16:55:17.221+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-28 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:17.221+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:17.524+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:17.524+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:17-54999e759a8f659f34bfb362", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353717524), what: "moveChunk.from", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 966, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 409, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:17.580+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7000df49298418feda') } ntoreturn:1 keyUpdates:0 reslen:37 1744ms 2014-12-23T16:55:17.581+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:17.581+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7000df49298418feda') } 2014-12-23T16:55:18.188+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e759a8f659f34bfb363 2014-12-23T16:55:18.188+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:18-54999e769a8f659f34bfb364", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353718188), what: "moveChunk.start", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:18.254+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 2|0||54999e7000df49298418feda, current metadata version is 2|0||54999e7000df49298418feda 2014-12-23T16:55:18.255+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-28 from shard version 2|0||54999e7000df49298418feda to shard version 2|1||54999e7000df49298418feda 2014-12-23T16:55:18.255+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e7000df49298418feda, took 0ms 2014-12-23T16:55:18.255+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e7000df49298418feda 2014-12-23T16:55:18.255+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:18.257+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:18.260+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:18.264+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:18.273+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:18.289+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:18.289+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:18.289+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:18.289+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e7000df49298418feda 2014-12-23T16:55:18.343+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:18.343+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e7000df49298418feda through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-28' 2014-12-23T16:55:18.647+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:18-54999e769a8f659f34bfb365", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353718647), what: "moveChunk.commit", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:18.703+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:18.703+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:18.703+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-28 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 851 2014-12-23T16:55:18.703+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-28 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:18.703+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:19.006+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:19.006+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:19-54999e779a8f659f34bfb366", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353719006), what: "moveChunk.from", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 414, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:19.057+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7000df49298418feda') } ntoreturn:1 keyUpdates:0 reslen:37 1475ms 2014-12-23T16:55:19.059+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-28", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-28-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e7000df49298418feda') } 2014-12-23T16:55:19.666+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e779a8f659f34bfb367 2014-12-23T16:55:19.666+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 3|0||54999e7000df49298418feda, current metadata version is 3|0||54999e7000df49298418feda 2014-12-23T16:55:19.667+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-28 from shard version 3|0||54999e7000df49298418feda to shard version 3|1||54999e7000df49298418feda 2014-12-23T16:55:19.667+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e7000df49298418feda, took 0ms 2014-12-23T16:55:19.667+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e7000df49298418feda 2014-12-23T16:55:19.945+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:19-54999e779a8f659f34bfb368", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353719945), what: "split", ns: "benchdb1.COL-28", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e7000df49298418feda') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e7000df49298418feda') } } } 2014-12-23T16:55:20.330+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:20.330+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-28", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-28-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e7000df49298418feda') } ntoreturn:1 keyUpdates:0 reslen:37 1271ms 2014-12-23T16:55:21.897+0000 I INDEX [conn2] build index on: benchdb1.COL-29 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-29" } 2014-12-23T16:55:21.897+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:21.898+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:22.217+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-29 with requested shard version 1|2||54999e7600df49298418fede, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:22.218+0000 I SHARDING [conn2] collection benchdb1.COL-29 was previously unsharded, new metadata loaded with shard version 1|2||54999e7600df49298418fede 2014-12-23T16:55:22.218+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e7600df49298418fede, took 0ms 2014-12-23T16:55:22.256+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:22.257+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7600df49298418fede') } 2014-12-23T16:55:22.879+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e7a9a8f659f34bfb36a 2014-12-23T16:55:22.879+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:22-54999e7a9a8f659f34bfb36b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353722879), what: "moveChunk.start", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:22.940+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 1|2||54999e7600df49298418fede, current metadata version is 1|2||54999e7600df49298418fede 2014-12-23T16:55:22.941+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-29 already up to date (shard version : 1|2||54999e7600df49298418fede, took 0ms) 2014-12-23T16:55:22.941+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e7600df49298418fede 2014-12-23T16:55:22.941+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:22.944+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:22.946+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:22.950+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:22.950+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:22.951+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:22.951+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e7600df49298418fede 2014-12-23T16:55:22.998+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:22.998+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e7600df49298418fede through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-29' 2014-12-23T16:55:23.303+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:23-54999e7b9a8f659f34bfb36c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353723303), what: "moveChunk.commit", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:23.361+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:23.361+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:23.361+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-29 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 872 2014-12-23T16:55:23.361+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-29 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:23.361+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:23.664+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:23.664+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:23-54999e7b9a8f659f34bfb36d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353723664), what: "moveChunk.from", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 684, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 410, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:23.713+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7600df49298418fede') } ntoreturn:1 keyUpdates:0 reslen:37 1457ms 2014-12-23T16:55:23.715+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:23.715+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7600df49298418fede') } 2014-12-23T16:55:24.322+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e7b9a8f659f34bfb36e 2014-12-23T16:55:24.322+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:24-54999e7c9a8f659f34bfb36f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353724322), what: "moveChunk.start", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:24.381+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 2|0||54999e7600df49298418fede, current metadata version is 2|0||54999e7600df49298418fede 2014-12-23T16:55:24.382+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-29 from shard version 2|0||54999e7600df49298418fede to shard version 2|1||54999e7600df49298418fede 2014-12-23T16:55:24.382+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e7600df49298418fede, took 0ms 2014-12-23T16:55:24.382+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e7600df49298418fede 2014-12-23T16:55:24.382+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:24.384+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:24.386+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:24.391+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:24.399+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:24.416+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:24.416+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:24.416+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:24.416+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e7600df49298418fede 2014-12-23T16:55:24.465+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:24.465+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e7600df49298418fede through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-29' 2014-12-23T16:55:24.769+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:24-54999e7c9a8f659f34bfb370", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353724769), what: "moveChunk.commit", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:24.819+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:24.819+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:24.819+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-29 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 880 2014-12-23T16:55:24.819+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-29 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:24.819+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:25.122+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:25.122+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:25-54999e7d9a8f659f34bfb371", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353725122), what: "moveChunk.from", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 403, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:25.170+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7600df49298418fede') } ntoreturn:1 keyUpdates:0 reslen:37 1454ms 2014-12-23T16:55:25.172+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-29", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-29-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e7600df49298418fede') } 2014-12-23T16:55:25.778+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e7d9a8f659f34bfb372 2014-12-23T16:55:25.778+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 3|0||54999e7600df49298418fede, current metadata version is 3|0||54999e7600df49298418fede 2014-12-23T16:55:25.779+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-29 from shard version 3|0||54999e7600df49298418fede to shard version 3|1||54999e7600df49298418fede 2014-12-23T16:55:25.779+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e7600df49298418fede, took 0ms 2014-12-23T16:55:25.779+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e7600df49298418fede 2014-12-23T16:55:26.083+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:26-54999e7e9a8f659f34bfb373", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353726083), what: "split", ns: "benchdb1.COL-29", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e7600df49298418fede') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e7600df49298418fede') } } } 2014-12-23T16:55:26.468+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:26.468+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-29", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-29-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e7600df49298418fede') } ntoreturn:1 keyUpdates:0 reslen:37 1296ms 2014-12-23T16:55:29.056+0000 I INDEX [conn2] build index on: benchdb1.COL-30 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-30" } 2014-12-23T16:55:29.056+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:29.057+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:29.323+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-30 with requested shard version 1|2||54999e7e00df49298418fee2, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:29.324+0000 I SHARDING [conn2] collection benchdb1.COL-30 was previously unsharded, new metadata loaded with shard version 1|2||54999e7e00df49298418fee2 2014-12-23T16:55:29.324+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e7e00df49298418fee2, took 0ms 2014-12-23T16:55:29.377+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:29.377+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7e00df49298418fee2') } 2014-12-23T16:55:30.251+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e819a8f659f34bfb375 2014-12-23T16:55:30.252+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:30-54999e829a8f659f34bfb376", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353730252), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:30.309+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|2||54999e7e00df49298418fee2, current metadata version is 1|2||54999e7e00df49298418fee2 2014-12-23T16:55:30.309+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|2||54999e7e00df49298418fee2, took 0ms) 2014-12-23T16:55:30.309+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e7e00df49298418fee2 2014-12-23T16:55:30.310+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:30.313+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:30.315+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:30.319+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:30.319+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:30.320+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:30.320+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e7e00df49298418fee2 2014-12-23T16:55:30.373+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:30.373+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e7e00df49298418fee2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-30' 2014-12-23T16:55:30.678+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:30-54999e829a8f659f34bfb377", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353730678), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:30.734+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:30.734+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:30.734+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-30 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 899 2014-12-23T16:55:30.734+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:30.734+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:31.037+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:31.037+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:31-54999e839a8f659f34bfb378", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353731037), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 932, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:31.089+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7e00df49298418fee2') } ntoreturn:1 keyUpdates:0 reslen:37 1711ms 2014-12-23T16:55:31.090+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:31.091+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7e00df49298418fee2') } 2014-12-23T16:55:31.698+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e839a8f659f34bfb379 2014-12-23T16:55:31.698+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:31-54999e839a8f659f34bfb37a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353731698), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:31.750+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 2|0||54999e7e00df49298418fee2, current metadata version is 2|0||54999e7e00df49298418fee2 2014-12-23T16:55:31.751+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-30 from shard version 2|0||54999e7e00df49298418fee2 to shard version 2|1||54999e7e00df49298418fee2 2014-12-23T16:55:31.751+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e7e00df49298418fee2, took 0ms 2014-12-23T16:55:31.751+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e7e00df49298418fee2 2014-12-23T16:55:31.751+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:31.753+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:31.756+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:31.760+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:31.769+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:31.785+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:31.785+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:31.785+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:31.785+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e7e00df49298418fee2 2014-12-23T16:55:31.839+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:31.839+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e7e00df49298418fee2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-30' 2014-12-23T16:55:32.144+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:32-54999e849a8f659f34bfb37b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353732144), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:32.204+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:32.204+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:32.204+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-30 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 907 2014-12-23T16:55:32.204+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:32.204+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:32.507+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:32.507+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:32-54999e849a8f659f34bfb37c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353732507), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 660, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 418, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:32.563+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e7e00df49298418fee2') } ntoreturn:1 keyUpdates:0 reslen:37 1473ms 2014-12-23T16:55:32.566+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e7e00df49298418fee2') } 2014-12-23T16:55:33.036+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e849a8f659f34bfb37d 2014-12-23T16:55:33.036+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 3|0||54999e7e00df49298418fee2, current metadata version is 3|0||54999e7e00df49298418fee2 2014-12-23T16:55:33.037+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-30 from shard version 3|0||54999e7e00df49298418fee2 to shard version 3|1||54999e7e00df49298418fee2 2014-12-23T16:55:33.037+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e7e00df49298418fee2, took 0ms 2014-12-23T16:55:33.037+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e7e00df49298418fee2 2014-12-23T16:55:33.232+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:33-54999e859a8f659f34bfb37e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353733232), what: "split", ns: "benchdb1.COL-30", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e7e00df49298418fee2') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e7e00df49298418fee2') } } } 2014-12-23T16:55:33.450+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:33.450+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e7e00df49298418fee2') } ntoreturn:1 keyUpdates:0 reslen:37 884ms 2014-12-23T16:55:34.034+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:55:33 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:55:35.245+0000 I INDEX [conn2] build index on: benchdb1.COL-31 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-31" } 2014-12-23T16:55:35.245+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:35.246+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:35.570+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-31 with requested shard version 1|2||54999e8400df49298418fee7, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:35.571+0000 I SHARDING [conn2] collection benchdb1.COL-31 was previously unsharded, new metadata loaded with shard version 1|2||54999e8400df49298418fee7 2014-12-23T16:55:35.571+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e8400df49298418fee7, took 0ms 2014-12-23T16:55:35.617+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:35.617+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8400df49298418fee7') } 2014-12-23T16:55:36.478+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e879a8f659f34bfb380 2014-12-23T16:55:36.478+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:36-54999e889a8f659f34bfb381", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353736478), what: "moveChunk.start", ns: "benchdb1.COL-31", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:36.550+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-31 based on current shard version 1|2||54999e8400df49298418fee7, current metadata version is 1|2||54999e8400df49298418fee7 2014-12-23T16:55:36.551+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-31 already up to date (shard version : 1|2||54999e8400df49298418fee7, took 0ms) 2014-12-23T16:55:36.551+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e8400df49298418fee7 2014-12-23T16:55:36.552+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:36.554+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:36.557+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:36.561+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:36.561+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:36.561+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:36.561+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e8400df49298418fee7 2014-12-23T16:55:36.613+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:36.613+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e8400df49298418fee7 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-31' 2014-12-23T16:55:36.918+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:36-54999e889a8f659f34bfb382", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353736918), what: "moveChunk.commit", ns: "benchdb1.COL-31", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:36.978+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:36.978+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:36.978+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-31 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 928 2014-12-23T16:55:36.978+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-31 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:36.978+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:37.281+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:37.281+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:37-54999e899a8f659f34bfb383", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353737281), what: "moveChunk.from", ns: "benchdb1.COL-31", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 934, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 417, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:37.329+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8400df49298418fee7') } ntoreturn:1 keyUpdates:0 reslen:37 1711ms 2014-12-23T16:55:37.330+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:37.331+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8400df49298418fee7') } 2014-12-23T16:55:37.937+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e899a8f659f34bfb384 2014-12-23T16:55:37.937+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:37-54999e899a8f659f34bfb385", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353737937), what: "moveChunk.start", ns: "benchdb1.COL-31", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:37.998+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-31 based on current shard version 2|0||54999e8400df49298418fee7, current metadata version is 2|0||54999e8400df49298418fee7 2014-12-23T16:55:37.999+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-31 from shard version 2|0||54999e8400df49298418fee7 to shard version 2|1||54999e8400df49298418fee7 2014-12-23T16:55:37.999+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e8400df49298418fee7, took 0ms 2014-12-23T16:55:37.999+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e8400df49298418fee7 2014-12-23T16:55:37.999+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:38.002+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:38.004+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:38.008+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:38.017+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:38.033+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:38.033+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:38.033+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:38.033+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e8400df49298418fee7 2014-12-23T16:55:38.083+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:38.083+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e8400df49298418fee7 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-31' 2014-12-23T16:55:38.387+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:38-54999e8a9a8f659f34bfb386", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353738387), what: "moveChunk.commit", ns: "benchdb1.COL-31", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:38.438+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:38.438+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:38.438+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-31 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 936 2014-12-23T16:55:38.438+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-31 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:38.438+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:38.741+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:38.741+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:38-54999e8a9a8f659f34bfb387", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353738741), what: "moveChunk.from", ns: "benchdb1.COL-31", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 405, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:38.789+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8400df49298418fee7') } ntoreturn:1 keyUpdates:0 reslen:37 1458ms 2014-12-23T16:55:38.791+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-31", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-31-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e8400df49298418fee7') } 2014-12-23T16:55:39.397+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e8a9a8f659f34bfb388 2014-12-23T16:55:39.397+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-31 based on current shard version 3|0||54999e8400df49298418fee7, current metadata version is 3|0||54999e8400df49298418fee7 2014-12-23T16:55:39.398+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-31 from shard version 3|0||54999e8400df49298418fee7 to shard version 3|1||54999e8400df49298418fee7 2014-12-23T16:55:39.398+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e8400df49298418fee7, took 0ms 2014-12-23T16:55:39.398+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e8400df49298418fee7 2014-12-23T16:55:39.703+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:39-54999e8b9a8f659f34bfb389", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353739703), what: "split", ns: "benchdb1.COL-31", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e8400df49298418fee7') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e8400df49298418fee7') } } } 2014-12-23T16:55:40.092+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:40.092+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-31", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-31-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e8400df49298418fee7') } ntoreturn:1 keyUpdates:0 reslen:37 1301ms 2014-12-23T16:55:42.540+0000 I INDEX [conn2] build index on: benchdb1.COL-32 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-32" } 2014-12-23T16:55:42.540+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:42.541+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:42.816+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-32 with requested shard version 1|2||54999e8b00df49298418feea, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:42.816+0000 I SHARDING [conn2] collection benchdb1.COL-32 was previously unsharded, new metadata loaded with shard version 1|2||54999e8b00df49298418feea 2014-12-23T16:55:42.816+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e8b00df49298418feea, took 0ms 2014-12-23T16:55:42.860+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:42.860+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8b00df49298418feea') } 2014-12-23T16:55:43.769+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e8f9a8f659f34bfb38b 2014-12-23T16:55:43.769+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:43-54999e8f9a8f659f34bfb38c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353743769), what: "moveChunk.start", ns: "benchdb1.COL-32", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:43.831+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-32 based on current shard version 1|2||54999e8b00df49298418feea, current metadata version is 1|2||54999e8b00df49298418feea 2014-12-23T16:55:43.832+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-32 already up to date (shard version : 1|2||54999e8b00df49298418feea, took 0ms) 2014-12-23T16:55:43.832+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e8b00df49298418feea 2014-12-23T16:55:43.832+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:43.835+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:43.837+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:43.841+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:43.841+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:43.842+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:43.842+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e8b00df49298418feea 2014-12-23T16:55:43.921+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:43.921+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e8b00df49298418feea through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-32' 2014-12-23T16:55:44.225+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:44-54999e909a8f659f34bfb38d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353744225), what: "moveChunk.commit", ns: "benchdb1.COL-32", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:44.282+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:44.283+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:44.283+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-32 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 955 2014-12-23T16:55:44.283+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-32 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:44.283+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:44.586+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:44.586+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:44-54999e909a8f659f34bfb38e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353744586), what: "moveChunk.from", ns: "benchdb1.COL-32", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 971, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 441, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:44.638+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8b00df49298418feea') } ntoreturn:1 keyUpdates:0 reslen:37 1778ms 2014-12-23T16:55:44.640+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:44.640+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8b00df49298418feea') } 2014-12-23T16:55:45.107+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e909a8f659f34bfb38f 2014-12-23T16:55:45.107+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:45-54999e919a8f659f34bfb390", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353745107), what: "moveChunk.start", ns: "benchdb1.COL-32", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:45.166+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-32 based on current shard version 2|0||54999e8b00df49298418feea, current metadata version is 2|0||54999e8b00df49298418feea 2014-12-23T16:55:45.167+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-32 from shard version 2|0||54999e8b00df49298418feea to shard version 2|1||54999e8b00df49298418feea 2014-12-23T16:55:45.167+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e8b00df49298418feea, took 0ms 2014-12-23T16:55:45.167+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e8b00df49298418feea 2014-12-23T16:55:45.167+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:45.169+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:45.172+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:45.176+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:45.185+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:45.201+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:45.201+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:45.201+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:45.201+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e8b00df49298418feea 2014-12-23T16:55:45.247+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:45.247+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e8b00df49298418feea through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-32' 2014-12-23T16:55:45.437+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:45-54999e919a8f659f34bfb391", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353745437), what: "moveChunk.commit", ns: "benchdb1.COL-32", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:45.488+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:45.488+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:45.488+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-32 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 963 2014-12-23T16:55:45.488+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-32 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:45.488+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:45.650+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:45.650+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:45-54999e919a8f659f34bfb392", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353745650), what: "moveChunk.from", ns: "benchdb1.COL-32", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 526, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 286, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:45.742+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e8b00df49298418feea') } ntoreturn:1 keyUpdates:0 reslen:37 1102ms 2014-12-23T16:55:45.744+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-32", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-32-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e8b00df49298418feea') } 2014-12-23T16:55:46.023+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e919a8f659f34bfb393 2014-12-23T16:55:46.023+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-32 based on current shard version 3|0||54999e8b00df49298418feea, current metadata version is 3|0||54999e8b00df49298418feea 2014-12-23T16:55:46.024+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-32 from shard version 3|0||54999e8b00df49298418feea to shard version 3|1||54999e8b00df49298418feea 2014-12-23T16:55:46.024+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e8b00df49298418feea, took 0ms 2014-12-23T16:55:46.024+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e8b00df49298418feea 2014-12-23T16:55:46.190+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:46-54999e929a8f659f34bfb394", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353746190), what: "split", ns: "benchdb1.COL-32", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e8b00df49298418feea') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e8b00df49298418feea') } } } 2014-12-23T16:55:46.401+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:46.402+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-32", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-32-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e8b00df49298418feea') } ntoreturn:1 keyUpdates:0 reslen:37 657ms 2014-12-23T16:55:48.906+0000 I INDEX [conn2] build index on: benchdb1.COL-33 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-33" } 2014-12-23T16:55:48.906+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:48.906+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:49.189+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-33 with requested shard version 1|2||54999e9100df49298418feef, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:49.190+0000 I SHARDING [conn2] collection benchdb1.COL-33 was previously unsharded, new metadata loaded with shard version 1|2||54999e9100df49298418feef 2014-12-23T16:55:49.190+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e9100df49298418feef, took 0ms 2014-12-23T16:55:49.233+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:49.234+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9100df49298418feef') } 2014-12-23T16:55:50.103+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e959a8f659f34bfb396 2014-12-23T16:55:50.103+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:50-54999e969a8f659f34bfb397", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353750103), what: "moveChunk.start", ns: "benchdb1.COL-33", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:50.158+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-33 based on current shard version 1|2||54999e9100df49298418feef, current metadata version is 1|2||54999e9100df49298418feef 2014-12-23T16:55:50.159+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-33 already up to date (shard version : 1|2||54999e9100df49298418feef, took 0ms) 2014-12-23T16:55:50.159+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e9100df49298418feef 2014-12-23T16:55:50.159+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:50.162+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:50.164+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:50.169+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:50.169+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:50.169+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:50.169+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e9100df49298418feef 2014-12-23T16:55:50.219+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:50.219+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e9100df49298418feef through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-33' 2014-12-23T16:55:50.524+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:50-54999e969a8f659f34bfb398", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353750524), what: "moveChunk.commit", ns: "benchdb1.COL-33", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:50.583+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:50.583+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:50.583+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-33 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 983 2014-12-23T16:55:50.583+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-33 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:50.583+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:50.850+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:50.851+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:50-54999e969a8f659f34bfb399", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353750850), what: "moveChunk.from", ns: "benchdb1.COL-33", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 925, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 413, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:50.900+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9100df49298418feef') } ntoreturn:1 keyUpdates:0 reslen:37 1666ms 2014-12-23T16:55:50.901+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:50.902+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9100df49298418feef') } 2014-12-23T16:55:51.508+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e969a8f659f34bfb39a 2014-12-23T16:55:51.508+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:51-54999e979a8f659f34bfb39b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353751508), what: "moveChunk.start", ns: "benchdb1.COL-33", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:51.564+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-33 based on current shard version 2|0||54999e9100df49298418feef, current metadata version is 2|0||54999e9100df49298418feef 2014-12-23T16:55:51.565+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-33 from shard version 2|0||54999e9100df49298418feef to shard version 2|1||54999e9100df49298418feef 2014-12-23T16:55:51.565+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e9100df49298418feef, took 0ms 2014-12-23T16:55:51.565+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e9100df49298418feef 2014-12-23T16:55:51.565+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:51.567+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:51.569+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:51.574+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:51.582+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:51.599+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:51.599+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:51.599+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:51.599+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e9100df49298418feef 2014-12-23T16:55:51.652+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:51.652+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e9100df49298418feef through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-33' 2014-12-23T16:55:51.957+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:51-54999e979a8f659f34bfb39c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353751957), what: "moveChunk.commit", ns: "benchdb1.COL-33", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:52.050+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:52.050+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:52.050+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-33 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 991 2014-12-23T16:55:52.050+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-33 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:52.050+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:52.353+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:52.353+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:52-54999e989a8f659f34bfb39d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353752353), what: "moveChunk.from", ns: "benchdb1.COL-33", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 451, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:52.401+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9100df49298418feef') } ntoreturn:1 keyUpdates:0 reslen:37 1499ms 2014-12-23T16:55:52.403+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-33", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-33-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e9100df49298418feef') } 2014-12-23T16:55:53.009+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e989a8f659f34bfb39e 2014-12-23T16:55:53.009+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-33 based on current shard version 3|0||54999e9100df49298418feef, current metadata version is 3|0||54999e9100df49298418feef 2014-12-23T16:55:53.010+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-33 from shard version 3|0||54999e9100df49298418feef to shard version 3|1||54999e9100df49298418feef 2014-12-23T16:55:53.010+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e9100df49298418feef, took 0ms 2014-12-23T16:55:53.010+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e9100df49298418feef 2014-12-23T16:55:53.314+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:53-54999e999a8f659f34bfb39f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353753314), what: "split", ns: "benchdb1.COL-33", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e9100df49298418feef') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e9100df49298418feef') } } } 2014-12-23T16:55:53.704+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:53.704+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-33", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-33-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e9100df49298418feef') } ntoreturn:1 keyUpdates:0 reslen:37 1301ms 2014-12-23T16:55:56.319+0000 I INDEX [conn2] build index on: benchdb1.COL-34 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-34" } 2014-12-23T16:55:56.319+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:55:56.320+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:55:56.601+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-34 with requested shard version 1|2||54999e9900df49298418fef2, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:55:56.601+0000 I SHARDING [conn2] collection benchdb1.COL-34 was previously unsharded, new metadata loaded with shard version 1|2||54999e9900df49298418fef2 2014-12-23T16:55:56.601+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e9900df49298418fef2, took 0ms 2014-12-23T16:55:56.652+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:56.653+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9900df49298418fef2') } 2014-12-23T16:55:57.293+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e9c9a8f659f34bfb3a1 2014-12-23T16:55:57.293+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:57-54999e9d9a8f659f34bfb3a2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353757293), what: "moveChunk.start", ns: "benchdb1.COL-34", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:55:57.346+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-34 based on current shard version 1|2||54999e9900df49298418fef2, current metadata version is 1|2||54999e9900df49298418fef2 2014-12-23T16:55:57.347+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-34 already up to date (shard version : 1|2||54999e9900df49298418fef2, took 0ms) 2014-12-23T16:55:57.347+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999e9900df49298418fef2 2014-12-23T16:55:57.347+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:57.350+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:57.352+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:57.357+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:57.357+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:57.357+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:57.357+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999e9900df49298418fef2 2014-12-23T16:55:57.401+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:57.402+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999e9900df49298418fef2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-34' 2014-12-23T16:55:57.594+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:57-54999e9d9a8f659f34bfb3a3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353757594), what: "moveChunk.commit", ns: "benchdb1.COL-34", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:57.643+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:57.643+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:57.643+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-34 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1011 2014-12-23T16:55:57.643+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-34 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:55:57.643+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:57.803+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:57.803+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:57-54999e9d9a8f659f34bfb3a4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353757803), what: "moveChunk.from", ns: "benchdb1.COL-34", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 693, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 286, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:55:57.856+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9900df49298418fef2') } ntoreturn:1 keyUpdates:0 reslen:37 1203ms 2014-12-23T16:55:57.858+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:55:57.858+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9900df49298418fef2') } 2014-12-23T16:55:58.136+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e9d9a8f659f34bfb3a5 2014-12-23T16:55:58.136+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:58-54999e9e9a8f659f34bfb3a6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353758136), what: "moveChunk.start", ns: "benchdb1.COL-34", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:55:58.186+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-34 based on current shard version 2|0||54999e9900df49298418fef2, current metadata version is 2|0||54999e9900df49298418fef2 2014-12-23T16:55:58.187+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-34 from shard version 2|0||54999e9900df49298418fef2 to shard version 2|1||54999e9900df49298418fef2 2014-12-23T16:55:58.187+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999e9900df49298418fef2, took 0ms 2014-12-23T16:55:58.187+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999e9900df49298418fef2 2014-12-23T16:55:58.187+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:55:58.189+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:58.192+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:58.196+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:58.205+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:58.221+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:55:58.221+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:55:58.221+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:55:58.222+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999e9900df49298418fef2 2014-12-23T16:55:58.284+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:55:58.284+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999e9900df49298418fef2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-34' 2014-12-23T16:55:58.368+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('54999e9900df49298418fef2') } 2014-12-23T16:55:58.369+0000 W SHARDING [conn4] migration already in progress 2014-12-23T16:55:58.369+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:58-54999e9e9a8f659f34bfb3a7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353758369), what: "moveChunk.from", ns: "benchdb1.COL-34", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T16:55:58.453+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:58-54999e9e9a8f659f34bfb3a8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353758453), what: "moveChunk.commit", ns: "benchdb1.COL-34", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:55:58.498+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:58.498+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:55:58.498+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-34 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1020 2014-12-23T16:55:58.498+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-34 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:55:58.498+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:55:58.705+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:55:58.705+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:58-54999e9e9a8f659f34bfb3a9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353758705), what: "moveChunk.from", ns: "benchdb1.COL-34", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 329, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 276, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:55:58.755+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9900df49298418fef2') } ntoreturn:1 keyUpdates:0 reslen:37 897ms 2014-12-23T16:55:58.757+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-34", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-34-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e9900df49298418fef2') } 2014-12-23T16:55:59.363+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999e9e9a8f659f34bfb3aa 2014-12-23T16:55:59.363+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-34 based on current shard version 3|0||54999e9900df49298418fef2, current metadata version is 3|0||54999e9900df49298418fef2 2014-12-23T16:55:59.364+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-34 from shard version 3|0||54999e9900df49298418fef2 to shard version 3|1||54999e9900df49298418fef2 2014-12-23T16:55:59.364+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999e9900df49298418fef2, took 0ms 2014-12-23T16:55:59.364+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999e9900df49298418fef2 2014-12-23T16:55:59.662+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:55:59-54999e9f9a8f659f34bfb3ab", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353759662), what: "split", ns: "benchdb1.COL-34", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e9900df49298418fef2') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e9900df49298418fef2') } } } 2014-12-23T16:56:00.050+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:00.050+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-34", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-34-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e9900df49298418fef2') } ntoreturn:1 keyUpdates:0 reslen:37 1293ms 2014-12-23T16:56:02.694+0000 I INDEX [conn2] build index on: benchdb1.COL-35 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-35" } 2014-12-23T16:56:02.694+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:02.695+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:03.004+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-35 with requested shard version 1|2||54999e9f00df49298418fef7, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:03.005+0000 I SHARDING [conn2] collection benchdb1.COL-35 was previously unsharded, new metadata loaded with shard version 1|2||54999e9f00df49298418fef7 2014-12-23T16:56:03.005+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999e9f00df49298418fef7, took 0ms 2014-12-23T16:56:03.051+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:03.051+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9f00df49298418fef7') } 2014-12-23T16:56:03.963+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ea39a8f659f34bfb3ad 2014-12-23T16:56:03.963+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:03-54999ea39a8f659f34bfb3ae", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353763963), what: "moveChunk.start", ns: "benchdb1.COL-35", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:04.023+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-35 based on current shard version 1|2||54999e9f00df49298418fef7, current metadata version is 1|2||54999e9f00df49298418fef7 2014-12-23T16:56:04.023+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-35 already up to date (shard version : 1|2||54999e9f00df49298418fef7, took 0ms) 2014-12-23T16:56:04.023+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999e9f00df49298418fef7 2014-12-23T16:56:04.024+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:04.026+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:04.029+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:04.033+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:04.033+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:04.034+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:04.034+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999e9f00df49298418fef7 2014-12-23T16:56:04.091+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:04.091+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999e9f00df49298418fef7 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-35' 2014-12-23T16:56:04.398+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:04-54999ea49a8f659f34bfb3af", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353764398), what: "moveChunk.commit", ns: "benchdb1.COL-35", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:04.468+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:04.468+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:04.468+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-35 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1039 2014-12-23T16:56:04.468+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-35 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:04.468+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:04.518+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:56:04 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:56:04.771+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:04.771+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:04-54999ea49a8f659f34bfb3b0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353764771), what: "moveChunk.from", ns: "benchdb1.COL-35", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 971, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 434, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:04.831+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9f00df49298418fef7') } ntoreturn:1 keyUpdates:0 reslen:37 1779ms 2014-12-23T16:56:04.832+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:04.833+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9f00df49298418fef7') } 2014-12-23T16:56:05.440+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ea49a8f659f34bfb3b1 2014-12-23T16:56:05.440+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:05-54999ea59a8f659f34bfb3b2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353765440), what: "moveChunk.start", ns: "benchdb1.COL-35", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:05.494+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-35 based on current shard version 2|0||54999e9f00df49298418fef7, current metadata version is 2|0||54999e9f00df49298418fef7 2014-12-23T16:56:05.494+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-35 from shard version 2|0||54999e9f00df49298418fef7 to shard version 2|1||54999e9f00df49298418fef7 2014-12-23T16:56:05.494+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999e9f00df49298418fef7, took 0ms 2014-12-23T16:56:05.494+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999e9f00df49298418fef7 2014-12-23T16:56:05.494+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:05.497+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:05.499+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:05.504+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:05.512+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:05.528+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:05.528+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:05.529+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:05.529+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999e9f00df49298418fef7 2014-12-23T16:56:05.579+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:05.579+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999e9f00df49298418fef7 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-35' 2014-12-23T16:56:05.879+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:05-54999ea59a8f659f34bfb3b3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353765879), what: "moveChunk.commit", ns: "benchdb1.COL-35", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:05.930+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:05.930+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:05.930+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-35 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1047 2014-12-23T16:56:05.930+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-35 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:05.930+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:06.184+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:06.184+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:06-54999ea69a8f659f34bfb3b4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353766184), what: "moveChunk.from", ns: "benchdb1.COL-35", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 661, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 401, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:06.249+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999e9f00df49298418fef7') } ntoreturn:1 keyUpdates:0 reslen:37 1417ms 2014-12-23T16:56:06.251+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-35", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-35-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e9f00df49298418fef7') } 2014-12-23T16:56:06.812+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ea69a8f659f34bfb3b5 2014-12-23T16:56:06.812+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-35 based on current shard version 3|0||54999e9f00df49298418fef7, current metadata version is 3|0||54999e9f00df49298418fef7 2014-12-23T16:56:06.813+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-35 from shard version 3|0||54999e9f00df49298418fef7 to shard version 3|1||54999e9f00df49298418fef7 2014-12-23T16:56:06.813+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999e9f00df49298418fef7, took 0ms 2014-12-23T16:56:06.813+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999e9f00df49298418fef7 2014-12-23T16:56:07.118+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:07-54999ea79a8f659f34bfb3b6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353767118), what: "split", ns: "benchdb1.COL-35", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999e9f00df49298418fef7') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999e9f00df49298418fef7') } } } 2014-12-23T16:56:07.487+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:07.487+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-35", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-35-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999e9f00df49298418fef7') } ntoreturn:1 keyUpdates:0 reslen:37 1235ms 2014-12-23T16:56:09.641+0000 I INDEX [conn2] build index on: benchdb1.COL-36 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-36" } 2014-12-23T16:56:09.641+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:09.642+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:09.921+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-36 with requested shard version 1|2||54999ea600df49298418fefb, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:09.922+0000 I SHARDING [conn2] collection benchdb1.COL-36 was previously unsharded, new metadata loaded with shard version 1|2||54999ea600df49298418fefb 2014-12-23T16:56:09.922+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ea600df49298418fefb, took 0ms 2014-12-23T16:56:09.966+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:09.966+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ea600df49298418fefb') } 2014-12-23T16:56:10.392+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eaa9a8f659f34bfb3b8 2014-12-23T16:56:10.392+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:10-54999eaa9a8f659f34bfb3b9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353770392), what: "moveChunk.start", ns: "benchdb1.COL-36", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:10.500+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-36 based on current shard version 1|2||54999ea600df49298418fefb, current metadata version is 1|2||54999ea600df49298418fefb 2014-12-23T16:56:10.501+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-36 already up to date (shard version : 1|2||54999ea600df49298418fefb, took 0ms) 2014-12-23T16:56:10.501+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999ea600df49298418fefb 2014-12-23T16:56:10.501+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:10.504+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:10.506+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:10.511+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:10.511+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:10.511+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:10.511+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999ea600df49298418fefb 2014-12-23T16:56:10.555+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:10.555+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999ea600df49298418fefb through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-36' 2014-12-23T16:56:10.750+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:10-54999eaa9a8f659f34bfb3ba", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353770750), what: "moveChunk.commit", ns: "benchdb1.COL-36", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:10.804+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:10.805+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:10.805+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-36 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1068 2014-12-23T16:56:10.805+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-36 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:10.805+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:10.977+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:10.977+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:10-54999eaa9a8f659f34bfb3bb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353770977), what: "moveChunk.from", ns: "benchdb1.COL-36", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 534, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 293, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:11.021+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ea600df49298418fefb') } ntoreturn:1 keyUpdates:0 reslen:37 1055ms 2014-12-23T16:56:11.022+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:11.023+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ea600df49298418fefb') } 2014-12-23T16:56:11.579+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eab9a8f659f34bfb3bc 2014-12-23T16:56:11.579+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:11-54999eab9a8f659f34bfb3bd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353771579), what: "moveChunk.start", ns: "benchdb1.COL-36", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:11.649+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-36 based on current shard version 2|0||54999ea600df49298418fefb, current metadata version is 2|0||54999ea600df49298418fefb 2014-12-23T16:56:11.649+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-36 from shard version 2|0||54999ea600df49298418fefb to shard version 2|1||54999ea600df49298418fefb 2014-12-23T16:56:11.649+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999ea600df49298418fefb, took 0ms 2014-12-23T16:56:11.649+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999ea600df49298418fefb 2014-12-23T16:56:11.649+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:11.652+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:11.654+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:11.659+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:11.667+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:11.683+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:11.683+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:11.684+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:11.684+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999ea600df49298418fefb 2014-12-23T16:56:11.734+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:11.734+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999ea600df49298418fefb through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-36' 2014-12-23T16:56:12.038+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:12-54999eac9a8f659f34bfb3be", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353772038), what: "moveChunk.commit", ns: "benchdb1.COL-36", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:12.089+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:12.089+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:12.089+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-36 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1076 2014-12-23T16:56:12.089+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-36 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:12.089+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:12.392+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:12.392+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:12-54999eac9a8f659f34bfb3bf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353772392), what: "moveChunk.from", ns: "benchdb1.COL-36", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 626, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 405, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:12.448+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ea600df49298418fefb') } ntoreturn:1 keyUpdates:0 reslen:37 1425ms 2014-12-23T16:56:12.450+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-36", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-36-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ea600df49298418fefb') } 2014-12-23T16:56:13.057+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eac9a8f659f34bfb3c0 2014-12-23T16:56:13.057+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-36 based on current shard version 3|0||54999ea600df49298418fefb, current metadata version is 3|0||54999ea600df49298418fefb 2014-12-23T16:56:13.058+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-36 from shard version 3|0||54999ea600df49298418fefb to shard version 3|1||54999ea600df49298418fefb 2014-12-23T16:56:13.058+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999ea600df49298418fefb, took 0ms 2014-12-23T16:56:13.058+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999ea600df49298418fefb 2014-12-23T16:56:13.363+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:13-54999ead9a8f659f34bfb3c1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353773363), what: "split", ns: "benchdb1.COL-36", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ea600df49298418fefb') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ea600df49298418fefb') } } } 2014-12-23T16:56:13.745+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:13.745+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-36", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-36-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ea600df49298418fefb') } ntoreturn:1 keyUpdates:0 reslen:37 1295ms 2014-12-23T16:56:16.337+0000 I INDEX [conn2] build index on: benchdb1.COL-37 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-37" } 2014-12-23T16:56:16.337+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:16.337+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:16.626+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-37 with requested shard version 1|2||54999ead00df49298418feff, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:16.626+0000 I SHARDING [conn2] collection benchdb1.COL-37 was previously unsharded, new metadata loaded with shard version 1|2||54999ead00df49298418feff 2014-12-23T16:56:16.626+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ead00df49298418feff, took 0ms 2014-12-23T16:56:16.674+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:16.674+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ead00df49298418feff') } 2014-12-23T16:56:17.583+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eb09a8f659f34bfb3c3 2014-12-23T16:56:17.583+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:17-54999eb19a8f659f34bfb3c4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353777583), what: "moveChunk.start", ns: "benchdb1.COL-37", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:17.639+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-37 based on current shard version 1|2||54999ead00df49298418feff, current metadata version is 1|2||54999ead00df49298418feff 2014-12-23T16:56:17.640+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-37 already up to date (shard version : 1|2||54999ead00df49298418feff, took 0ms) 2014-12-23T16:56:17.640+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999ead00df49298418feff 2014-12-23T16:56:17.640+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:17.643+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:17.645+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:17.649+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:17.649+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:17.650+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:17.650+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999ead00df49298418feff 2014-12-23T16:56:17.702+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:17.702+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999ead00df49298418feff through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-37' 2014-12-23T16:56:18.007+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:18-54999eb29a8f659f34bfb3c5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353778007), what: "moveChunk.commit", ns: "benchdb1.COL-37", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:18.116+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:18.116+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:18.116+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-37 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1095 2014-12-23T16:56:18.116+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-37 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:18.116+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:18.419+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:18.419+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:18-54999eb29a8f659f34bfb3c6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353778419), what: "moveChunk.from", ns: "benchdb1.COL-37", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 965, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 466, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:18.475+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ead00df49298418feff') } ntoreturn:1 keyUpdates:0 reslen:37 1800ms 2014-12-23T16:56:18.476+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:18.477+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ead00df49298418feff') } 2014-12-23T16:56:19.083+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eb29a8f659f34bfb3c7 2014-12-23T16:56:19.083+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:19-54999eb39a8f659f34bfb3c8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353779083), what: "moveChunk.start", ns: "benchdb1.COL-37", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:19.136+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-37 based on current shard version 2|0||54999ead00df49298418feff, current metadata version is 2|0||54999ead00df49298418feff 2014-12-23T16:56:19.137+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-37 from shard version 2|0||54999ead00df49298418feff to shard version 2|1||54999ead00df49298418feff 2014-12-23T16:56:19.137+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999ead00df49298418feff, took 0ms 2014-12-23T16:56:19.137+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999ead00df49298418feff 2014-12-23T16:56:19.137+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:19.139+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:19.142+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:19.146+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:19.154+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:19.171+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:19.171+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:19.171+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:19.171+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999ead00df49298418feff 2014-12-23T16:56:19.227+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:19.227+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999ead00df49298418feff through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-37' 2014-12-23T16:56:19.517+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:19-54999eb39a8f659f34bfb3c9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353779517), what: "moveChunk.commit", ns: "benchdb1.COL-37", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:19.572+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:19.572+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:19.572+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-37 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1103 2014-12-23T16:56:19.572+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-37 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:19.572+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:19.875+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:19.875+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:19-54999eb39a8f659f34bfb3ca", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353779875), what: "moveChunk.from", ns: "benchdb1.COL-37", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 660, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 401, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:19.931+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ead00df49298418feff') } ntoreturn:1 keyUpdates:0 reslen:37 1454ms 2014-12-23T16:56:19.933+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-37", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-37-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ead00df49298418feff') } 2014-12-23T16:56:20.540+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eb39a8f659f34bfb3cb 2014-12-23T16:56:20.540+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-37 based on current shard version 3|0||54999ead00df49298418feff, current metadata version is 3|0||54999ead00df49298418feff 2014-12-23T16:56:20.541+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-37 from shard version 3|0||54999ead00df49298418feff to shard version 3|1||54999ead00df49298418feff 2014-12-23T16:56:20.541+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999ead00df49298418feff, took 0ms 2014-12-23T16:56:20.541+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999ead00df49298418feff 2014-12-23T16:56:20.845+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:20-54999eb49a8f659f34bfb3cc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353780845), what: "split", ns: "benchdb1.COL-37", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ead00df49298418feff') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ead00df49298418feff') } } } 2014-12-23T16:56:21.227+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:21.227+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-37", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-37-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ead00df49298418feff') } ntoreturn:1 keyUpdates:0 reslen:37 1294ms 2014-12-23T16:56:22.824+0000 I INDEX [conn2] build index on: benchdb1.COL-38 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-38" } 2014-12-23T16:56:22.824+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:22.824+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:23.099+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-38 with requested shard version 1|2||54999eb300df49298418ff03, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:23.100+0000 I SHARDING [conn2] collection benchdb1.COL-38 was previously unsharded, new metadata loaded with shard version 1|2||54999eb300df49298418ff03 2014-12-23T16:56:23.100+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999eb300df49298418ff03, took 0ms 2014-12-23T16:56:23.143+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:23.144+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eb300df49298418ff03') } 2014-12-23T16:56:23.614+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eb79a8f659f34bfb3ce 2014-12-23T16:56:23.614+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:23-54999eb79a8f659f34bfb3cf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353783614), what: "moveChunk.start", ns: "benchdb1.COL-38", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:23.655+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-38 based on current shard version 1|2||54999eb300df49298418ff03, current metadata version is 1|2||54999eb300df49298418ff03 2014-12-23T16:56:23.656+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-38 already up to date (shard version : 1|2||54999eb300df49298418ff03, took 0ms) 2014-12-23T16:56:23.656+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999eb300df49298418ff03 2014-12-23T16:56:23.656+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:23.659+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:23.661+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:23.666+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:23.666+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:23.666+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:23.666+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999eb300df49298418ff03 2014-12-23T16:56:23.717+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:23.717+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999eb300df49298418ff03 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-38' 2014-12-23T16:56:24.021+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:24-54999eb89a8f659f34bfb3d0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353784021), what: "moveChunk.commit", ns: "benchdb1.COL-38", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:24.093+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:24.094+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:24.094+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-38 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1124 2014-12-23T16:56:24.094+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-38 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:24.094+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:24.397+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:24.397+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:24-54999eb89a8f659f34bfb3d1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353784397), what: "moveChunk.from", ns: "benchdb1.COL-38", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 511, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 427, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:24.447+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eb300df49298418ff03') } ntoreturn:1 keyUpdates:0 reslen:37 1303ms 2014-12-23T16:56:24.449+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:24.449+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eb300df49298418ff03') } 2014-12-23T16:56:25.056+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eb89a8f659f34bfb3d2 2014-12-23T16:56:25.056+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:25-54999eb99a8f659f34bfb3d3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353785056), what: "moveChunk.start", ns: "benchdb1.COL-38", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:25.120+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-38 based on current shard version 2|0||54999eb300df49298418ff03, current metadata version is 2|0||54999eb300df49298418ff03 2014-12-23T16:56:25.121+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-38 from shard version 2|0||54999eb300df49298418ff03 to shard version 2|1||54999eb300df49298418ff03 2014-12-23T16:56:25.121+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999eb300df49298418ff03, took 0ms 2014-12-23T16:56:25.121+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999eb300df49298418ff03 2014-12-23T16:56:25.121+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:25.124+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:25.126+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:25.130+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:25.139+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:25.155+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:25.155+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:25.156+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:25.156+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999eb300df49298418ff03 2014-12-23T16:56:25.210+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:25.210+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999eb300df49298418ff03 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-38' 2014-12-23T16:56:25.515+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:25-54999eb99a8f659f34bfb3d4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353785515), what: "moveChunk.commit", ns: "benchdb1.COL-38", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:25.567+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:25.567+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:25.567+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-38 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1132 2014-12-23T16:56:25.567+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-38 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:25.567+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:25.870+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:25.870+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:25-54999eb99a8f659f34bfb3d5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353785870), what: "moveChunk.from", ns: "benchdb1.COL-38", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 411, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:25.929+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eb300df49298418ff03') } ntoreturn:1 keyUpdates:0 reslen:37 1480ms 2014-12-23T16:56:25.931+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-38", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-38-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999eb300df49298418ff03') } 2014-12-23T16:56:26.538+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eb99a8f659f34bfb3d6 2014-12-23T16:56:26.538+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-38 based on current shard version 3|0||54999eb300df49298418ff03, current metadata version is 3|0||54999eb300df49298418ff03 2014-12-23T16:56:26.539+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-38 from shard version 3|0||54999eb300df49298418ff03 to shard version 3|1||54999eb300df49298418ff03 2014-12-23T16:56:26.539+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999eb300df49298418ff03, took 0ms 2014-12-23T16:56:26.539+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999eb300df49298418ff03 2014-12-23T16:56:26.843+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:26-54999eba9a8f659f34bfb3d7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353786843), what: "split", ns: "benchdb1.COL-38", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999eb300df49298418ff03') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999eb300df49298418ff03') } } } 2014-12-23T16:56:27.227+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:27.227+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-38", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-38-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999eb300df49298418ff03') } ntoreturn:1 keyUpdates:0 reslen:37 1295ms 2014-12-23T16:56:29.821+0000 I INDEX [conn2] build index on: benchdb1.COL-39 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-39" } 2014-12-23T16:56:29.821+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:29.822+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:30.140+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-39 with requested shard version 1|2||54999eba00df49298418ff07, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:30.141+0000 I SHARDING [conn2] collection benchdb1.COL-39 was previously unsharded, new metadata loaded with shard version 1|2||54999eba00df49298418ff07 2014-12-23T16:56:30.141+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999eba00df49298418ff07, took 0ms 2014-12-23T16:56:30.185+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:30.185+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eba00df49298418ff07') } 2014-12-23T16:56:31.095+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ebe9a8f659f34bfb3d9 2014-12-23T16:56:31.095+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:31-54999ebf9a8f659f34bfb3da", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353791095), what: "moveChunk.start", ns: "benchdb1.COL-39", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:31.155+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-39 based on current shard version 1|2||54999eba00df49298418ff07, current metadata version is 1|2||54999eba00df49298418ff07 2014-12-23T16:56:31.155+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-39 already up to date (shard version : 1|2||54999eba00df49298418ff07, took 0ms) 2014-12-23T16:56:31.155+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999eba00df49298418ff07 2014-12-23T16:56:31.156+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:31.158+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:31.161+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:31.165+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:31.165+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:31.165+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:31.165+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999eba00df49298418ff07 2014-12-23T16:56:31.218+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:31.218+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999eba00df49298418ff07 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-39' 2014-12-23T16:56:31.523+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:31-54999ebf9a8f659f34bfb3db", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353791523), what: "moveChunk.commit", ns: "benchdb1.COL-39", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:31.583+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:31.583+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:31.584+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-39 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1151 2014-12-23T16:56:31.584+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-39 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:31.584+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:31.887+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:31.887+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:31-54999ebf9a8f659f34bfb3dc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353791887), what: "moveChunk.from", ns: "benchdb1.COL-39", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 970, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 418, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:31.944+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eba00df49298418ff07') } ntoreturn:1 keyUpdates:0 reslen:37 1759ms 2014-12-23T16:56:31.946+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:31.946+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eba00df49298418ff07') } 2014-12-23T16:56:32.565+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ebf9a8f659f34bfb3dd 2014-12-23T16:56:32.565+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:32-54999ec09a8f659f34bfb3de", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353792565), what: "moveChunk.start", ns: "benchdb1.COL-39", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:32.618+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-39 based on current shard version 2|0||54999eba00df49298418ff07, current metadata version is 2|0||54999eba00df49298418ff07 2014-12-23T16:56:32.619+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-39 from shard version 2|0||54999eba00df49298418ff07 to shard version 2|1||54999eba00df49298418ff07 2014-12-23T16:56:32.619+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999eba00df49298418ff07, took 0ms 2014-12-23T16:56:32.619+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999eba00df49298418ff07 2014-12-23T16:56:32.619+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:32.621+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:32.624+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:32.628+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:32.637+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:32.653+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:32.653+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:32.654+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:32.654+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999eba00df49298418ff07 2014-12-23T16:56:32.709+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:32.709+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999eba00df49298418ff07 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-39' 2014-12-23T16:56:33.014+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:33-54999ec19a8f659f34bfb3df", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353793014), what: "moveChunk.commit", ns: "benchdb1.COL-39", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:33.063+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:33.063+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:33.064+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-39 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1159 2014-12-23T16:56:33.064+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-39 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:33.064+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:33.367+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:33.367+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:33-54999ec19a8f659f34bfb3e0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353793367), what: "moveChunk.from", ns: "benchdb1.COL-39", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 410, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:33.420+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999eba00df49298418ff07') } ntoreturn:1 keyUpdates:0 reslen:37 1474ms 2014-12-23T16:56:33.422+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-39", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-39-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999eba00df49298418ff07') } 2014-12-23T16:56:33.973+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ec19a8f659f34bfb3e1 2014-12-23T16:56:33.973+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-39 based on current shard version 3|0||54999eba00df49298418ff07, current metadata version is 3|0||54999eba00df49298418ff07 2014-12-23T16:56:33.973+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-39 from shard version 3|0||54999eba00df49298418ff07 to shard version 3|1||54999eba00df49298418ff07 2014-12-23T16:56:33.973+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999eba00df49298418ff07, took 0ms 2014-12-23T16:56:33.973+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999eba00df49298418ff07 2014-12-23T16:56:34.231+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:34-54999ec29a8f659f34bfb3e2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353794231), what: "split", ns: "benchdb1.COL-39", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999eba00df49298418ff07') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999eba00df49298418ff07') } } } 2014-12-23T16:56:34.527+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:34.527+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-39", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-39-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999eba00df49298418ff07') } ntoreturn:1 keyUpdates:0 reslen:37 1104ms 2014-12-23T16:56:34.796+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:56:34 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:56:35.905+0000 I INDEX [conn2] build index on: benchdb1.COL-40 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-40" } 2014-12-23T16:56:35.905+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:35.906+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:36.233+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-40 with requested shard version 1|2||54999ec000df49298418ff0c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:36.233+0000 I SHARDING [conn2] collection benchdb1.COL-40 was previously unsharded, new metadata loaded with shard version 1|2||54999ec000df49298418ff0c 2014-12-23T16:56:36.233+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ec000df49298418ff0c, took 0ms 2014-12-23T16:56:36.275+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:36.275+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec000df49298418ff0c') } 2014-12-23T16:56:37.070+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ec49a8f659f34bfb3e4 2014-12-23T16:56:37.070+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:37-54999ec59a8f659f34bfb3e5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353797070), what: "moveChunk.start", ns: "benchdb1.COL-40", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:37.146+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-40 based on current shard version 1|2||54999ec000df49298418ff0c, current metadata version is 1|2||54999ec000df49298418ff0c 2014-12-23T16:56:37.147+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-40 already up to date (shard version : 1|2||54999ec000df49298418ff0c, took 0ms) 2014-12-23T16:56:37.147+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999ec000df49298418ff0c 2014-12-23T16:56:37.147+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:37.150+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:37.153+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:37.157+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:37.157+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:37.157+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:37.157+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999ec000df49298418ff0c 2014-12-23T16:56:37.205+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:37.205+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999ec000df49298418ff0c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-40' 2014-12-23T16:56:37.510+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:37-54999ec59a8f659f34bfb3e6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353797510), what: "moveChunk.commit", ns: "benchdb1.COL-40", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:37.568+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:37.569+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:37.569+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-40 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1180 2014-12-23T16:56:37.569+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-40 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:37.569+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:37.872+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:37.872+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:37-54999ec59a8f659f34bfb3e7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353797872), what: "moveChunk.from", ns: "benchdb1.COL-40", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 871, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 411, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:37.923+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec000df49298418ff0c') } ntoreturn:1 keyUpdates:0 reslen:37 1647ms 2014-12-23T16:56:37.924+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:37.924+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec000df49298418ff0c') } 2014-12-23T16:56:38.532+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ec59a8f659f34bfb3e8 2014-12-23T16:56:38.532+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:38-54999ec69a8f659f34bfb3e9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353798532), what: "moveChunk.start", ns: "benchdb1.COL-40", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:38.586+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-40 based on current shard version 2|0||54999ec000df49298418ff0c, current metadata version is 2|0||54999ec000df49298418ff0c 2014-12-23T16:56:38.587+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-40 from shard version 2|0||54999ec000df49298418ff0c to shard version 2|1||54999ec000df49298418ff0c 2014-12-23T16:56:38.587+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999ec000df49298418ff0c, took 0ms 2014-12-23T16:56:38.587+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999ec000df49298418ff0c 2014-12-23T16:56:38.587+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:38.589+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:38.591+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:38.596+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:38.604+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:38.621+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:38.621+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:38.621+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:38.621+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999ec000df49298418ff0c 2014-12-23T16:56:38.670+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:38.670+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999ec000df49298418ff0c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-40' 2014-12-23T16:56:38.975+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:38-54999ec69a8f659f34bfb3ea", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353798975), what: "moveChunk.commit", ns: "benchdb1.COL-40", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:39.027+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:39.027+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:39.027+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-40 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1188 2014-12-23T16:56:39.027+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-40 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:39.027+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:39.330+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:39.330+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:39-54999ec79a8f659f34bfb3eb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353799330), what: "moveChunk.from", ns: "benchdb1.COL-40", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 0, step 4 of 6: 32, step 5 of 6: 406, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:39.407+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec000df49298418ff0c') } ntoreturn:1 keyUpdates:0 reslen:37 1482ms 2014-12-23T16:56:39.409+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-40", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-40-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ec000df49298418ff0c') } 2014-12-23T16:56:40.015+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ec79a8f659f34bfb3ec 2014-12-23T16:56:40.015+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-40 based on current shard version 3|0||54999ec000df49298418ff0c, current metadata version is 3|0||54999ec000df49298418ff0c 2014-12-23T16:56:40.016+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-40 from shard version 3|0||54999ec000df49298418ff0c to shard version 3|1||54999ec000df49298418ff0c 2014-12-23T16:56:40.016+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999ec000df49298418ff0c, took 0ms 2014-12-23T16:56:40.016+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999ec000df49298418ff0c 2014-12-23T16:56:40.321+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:40-54999ec89a8f659f34bfb3ed", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353800321), what: "split", ns: "benchdb1.COL-40", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ec000df49298418ff0c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ec000df49298418ff0c') } } } 2014-12-23T16:56:40.700+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:40.700+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-40", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-40-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ec000df49298418ff0c') } ntoreturn:1 keyUpdates:0 reslen:37 1290ms 2014-12-23T16:56:43.256+0000 I INDEX [conn2] build index on: benchdb1.COL-41 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-41" } 2014-12-23T16:56:43.256+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:43.257+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:43.541+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-41 with requested shard version 1|2||54999ec800df49298418ff0f, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:43.542+0000 I SHARDING [conn2] collection benchdb1.COL-41 was previously unsharded, new metadata loaded with shard version 1|2||54999ec800df49298418ff0f 2014-12-23T16:56:43.542+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ec800df49298418ff0f, took 0ms 2014-12-23T16:56:43.586+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:43.586+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec800df49298418ff0f') } 2014-12-23T16:56:44.498+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ecb9a8f659f34bfb3ef 2014-12-23T16:56:44.499+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:44-54999ecc9a8f659f34bfb3f0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353804499), what: "moveChunk.start", ns: "benchdb1.COL-41", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:44.563+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-41 based on current shard version 1|2||54999ec800df49298418ff0f, current metadata version is 1|2||54999ec800df49298418ff0f 2014-12-23T16:56:44.563+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-41 already up to date (shard version : 1|2||54999ec800df49298418ff0f, took 0ms) 2014-12-23T16:56:44.563+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999ec800df49298418ff0f 2014-12-23T16:56:44.564+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:44.566+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:44.569+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:44.573+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:44.573+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:44.574+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:44.574+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999ec800df49298418ff0f 2014-12-23T16:56:44.626+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:44.626+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999ec800df49298418ff0f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-41' 2014-12-23T16:56:44.931+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:44-54999ecc9a8f659f34bfb3f1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353804931), what: "moveChunk.commit", ns: "benchdb1.COL-41", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:44.985+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:44.985+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:44.985+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-41 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1207 2014-12-23T16:56:44.985+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-41 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:44.985+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:45.288+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:45.289+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:45-54999ecd9a8f659f34bfb3f2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353805289), what: "moveChunk.from", ns: "benchdb1.COL-41", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 976, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 411, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:45.348+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec800df49298418ff0f') } ntoreturn:1 keyUpdates:0 reslen:37 1762ms 2014-12-23T16:56:45.350+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:45.350+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec800df49298418ff0f') } 2014-12-23T16:56:45.957+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ecd9a8f659f34bfb3f3 2014-12-23T16:56:45.957+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:45-54999ecd9a8f659f34bfb3f4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353805957), what: "moveChunk.start", ns: "benchdb1.COL-41", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:46.011+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-41 based on current shard version 2|0||54999ec800df49298418ff0f, current metadata version is 2|0||54999ec800df49298418ff0f 2014-12-23T16:56:46.012+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-41 from shard version 2|0||54999ec800df49298418ff0f to shard version 2|1||54999ec800df49298418ff0f 2014-12-23T16:56:46.012+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999ec800df49298418ff0f, took 0ms 2014-12-23T16:56:46.012+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999ec800df49298418ff0f 2014-12-23T16:56:46.012+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:46.015+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:46.017+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:46.022+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:46.030+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:46.047+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:46.047+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:46.047+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:46.047+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999ec800df49298418ff0f 2014-12-23T16:56:46.102+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:46.102+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999ec800df49298418ff0f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-41' 2014-12-23T16:56:46.362+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:46-54999ece9a8f659f34bfb3f5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353806362), what: "moveChunk.commit", ns: "benchdb1.COL-41", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:46.432+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:46.432+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:46.432+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-41 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1215 2014-12-23T16:56:46.433+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-41 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:46.433+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:46.644+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:46.644+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:46-54999ece9a8f659f34bfb3f6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353806644), what: "moveChunk.from", ns: "benchdb1.COL-41", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 385, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:46.695+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ec800df49298418ff0f') } ntoreturn:1 keyUpdates:0 reslen:37 1345ms 2014-12-23T16:56:46.697+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-41", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-41-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ec800df49298418ff0f') } 2014-12-23T16:56:46.996+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ece9a8f659f34bfb3f7 2014-12-23T16:56:46.996+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-41 based on current shard version 3|0||54999ec800df49298418ff0f, current metadata version is 3|0||54999ec800df49298418ff0f 2014-12-23T16:56:46.997+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-41 from shard version 3|0||54999ec800df49298418ff0f to shard version 3|1||54999ec800df49298418ff0f 2014-12-23T16:56:46.997+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999ec800df49298418ff0f, took 0ms 2014-12-23T16:56:46.997+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999ec800df49298418ff0f 2014-12-23T16:56:47.163+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:47-54999ecf9a8f659f34bfb3f8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353807163), what: "split", ns: "benchdb1.COL-41", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ec800df49298418ff0f') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ec800df49298418ff0f') } } } 2014-12-23T16:56:47.429+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:47.429+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-41", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-41-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ec800df49298418ff0f') } ntoreturn:1 keyUpdates:0 reslen:37 732ms 2014-12-23T16:56:49.281+0000 I INDEX [conn2] build index on: benchdb1.COL-42 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-42" } 2014-12-23T16:56:49.281+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:49.282+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:49.554+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-42 with requested shard version 1|2||54999ece00df49298418ff14, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:49.555+0000 I SHARDING [conn2] collection benchdb1.COL-42 was previously unsharded, new metadata loaded with shard version 1|2||54999ece00df49298418ff14 2014-12-23T16:56:49.555+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ece00df49298418ff14, took 0ms 2014-12-23T16:56:49.600+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:49.600+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ece00df49298418ff14') } 2014-12-23T16:56:50.509+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ed19a8f659f34bfb3fa 2014-12-23T16:56:50.509+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:50-54999ed29a8f659f34bfb3fb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353810509), what: "moveChunk.start", ns: "benchdb1.COL-42", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:50.576+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-42 based on current shard version 1|2||54999ece00df49298418ff14, current metadata version is 1|2||54999ece00df49298418ff14 2014-12-23T16:56:50.577+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-42 already up to date (shard version : 1|2||54999ece00df49298418ff14, took 0ms) 2014-12-23T16:56:50.577+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999ece00df49298418ff14 2014-12-23T16:56:50.578+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:50.581+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:50.583+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:50.587+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:50.588+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:50.588+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:50.588+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999ece00df49298418ff14 2014-12-23T16:56:50.639+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:50.639+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999ece00df49298418ff14 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-42' 2014-12-23T16:56:50.943+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:50-54999ed29a8f659f34bfb3fc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353810943), what: "moveChunk.commit", ns: "benchdb1.COL-42", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:51.002+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:51.002+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:51.003+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-42 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1236 2014-12-23T16:56:51.003+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-42 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:51.003+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:51.305+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:51.305+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:51-54999ed39a8f659f34bfb3fd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353811305), what: "moveChunk.from", ns: "benchdb1.COL-42", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 976, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:51.355+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ece00df49298418ff14') } ntoreturn:1 keyUpdates:0 reslen:37 1755ms 2014-12-23T16:56:51.357+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:51.357+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ece00df49298418ff14') } 2014-12-23T16:56:51.964+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ed39a8f659f34bfb3fe 2014-12-23T16:56:51.964+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:51-54999ed39a8f659f34bfb3ff", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353811964), what: "moveChunk.start", ns: "benchdb1.COL-42", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:52.019+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-42 based on current shard version 2|0||54999ece00df49298418ff14, current metadata version is 2|0||54999ece00df49298418ff14 2014-12-23T16:56:52.020+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-42 from shard version 2|0||54999ece00df49298418ff14 to shard version 2|1||54999ece00df49298418ff14 2014-12-23T16:56:52.020+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999ece00df49298418ff14, took 0ms 2014-12-23T16:56:52.020+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999ece00df49298418ff14 2014-12-23T16:56:52.020+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:52.023+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:52.025+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:52.029+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:52.038+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:52.054+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:52.054+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:52.055+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:52.055+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999ece00df49298418ff14 2014-12-23T16:56:52.106+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:52.106+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999ece00df49298418ff14 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-42' 2014-12-23T16:56:52.411+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:52-54999ed49a8f659f34bfb400", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353812411), what: "moveChunk.commit", ns: "benchdb1.COL-42", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:52.471+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:52.471+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:52.471+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-42 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1244 2014-12-23T16:56:52.471+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-42 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:52.471+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:52.774+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:52.774+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:52-54999ed49a8f659f34bfb401", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353812774), what: "moveChunk.from", ns: "benchdb1.COL-42", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 416, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:52.824+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ece00df49298418ff14') } ntoreturn:1 keyUpdates:0 reslen:37 1467ms 2014-12-23T16:56:52.826+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-42", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-42-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ece00df49298418ff14') } 2014-12-23T16:56:53.432+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ed49a8f659f34bfb402 2014-12-23T16:56:53.432+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-42 based on current shard version 3|0||54999ece00df49298418ff14, current metadata version is 3|0||54999ece00df49298418ff14 2014-12-23T16:56:53.433+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-42 from shard version 3|0||54999ece00df49298418ff14 to shard version 3|1||54999ece00df49298418ff14 2014-12-23T16:56:53.433+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999ece00df49298418ff14, took 0ms 2014-12-23T16:56:53.433+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999ece00df49298418ff14 2014-12-23T16:56:53.737+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:53-54999ed59a8f659f34bfb403", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353813737), what: "split", ns: "benchdb1.COL-42", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ece00df49298418ff14') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ece00df49298418ff14') } } } 2014-12-23T16:56:54.123+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:54.123+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-42", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-42-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ece00df49298418ff14') } ntoreturn:1 keyUpdates:0 reslen:37 1297ms 2014-12-23T16:56:56.723+0000 I INDEX [conn2] build index on: benchdb1.COL-43 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-43" } 2014-12-23T16:56:56.723+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:56:56.724+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:56:57.054+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-43 with requested shard version 1|2||54999ed500df49298418ff17, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:56:57.055+0000 I SHARDING [conn2] collection benchdb1.COL-43 was previously unsharded, new metadata loaded with shard version 1|2||54999ed500df49298418ff17 2014-12-23T16:56:57.055+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ed500df49298418ff17, took 0ms 2014-12-23T16:56:57.134+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:57.135+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ed500df49298418ff17') } 2014-12-23T16:56:58.044+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ed99a8f659f34bfb405 2014-12-23T16:56:58.044+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:58-54999eda9a8f659f34bfb406", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353818044), what: "moveChunk.start", ns: "benchdb1.COL-43", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:56:58.105+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-43 based on current shard version 1|2||54999ed500df49298418ff17, current metadata version is 1|2||54999ed500df49298418ff17 2014-12-23T16:56:58.105+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-43 already up to date (shard version : 1|2||54999ed500df49298418ff17, took 0ms) 2014-12-23T16:56:58.105+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999ed500df49298418ff17 2014-12-23T16:56:58.106+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:58.108+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:58.111+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:58.115+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:58.115+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:58.116+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:58.116+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999ed500df49298418ff17 2014-12-23T16:56:58.170+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:58.170+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999ed500df49298418ff17 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-43' 2014-12-23T16:56:58.474+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:58-54999eda9a8f659f34bfb407", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353818474), what: "moveChunk.commit", ns: "benchdb1.COL-43", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:58.538+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:58.538+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:58.538+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-43 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1263 2014-12-23T16:56:58.538+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-43 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:56:58.538+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:58.791+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:58.791+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:58-54999eda9a8f659f34bfb408", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353818791), what: "moveChunk.from", ns: "benchdb1.COL-43", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 970, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 422, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:56:58.853+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ed500df49298418ff17') } ntoreturn:1 keyUpdates:0 reslen:37 1718ms 2014-12-23T16:56:58.854+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:56:58.854+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ed500df49298418ff17') } 2014-12-23T16:56:59.212+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eda9a8f659f34bfb409 2014-12-23T16:56:59.212+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:59-54999edb9a8f659f34bfb40a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353819212), what: "moveChunk.start", ns: "benchdb1.COL-43", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:56:59.264+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-43 based on current shard version 2|0||54999ed500df49298418ff17, current metadata version is 2|0||54999ed500df49298418ff17 2014-12-23T16:56:59.264+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-43 from shard version 2|0||54999ed500df49298418ff17 to shard version 2|1||54999ed500df49298418ff17 2014-12-23T16:56:59.264+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999ed500df49298418ff17, took 0ms 2014-12-23T16:56:59.264+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999ed500df49298418ff17 2014-12-23T16:56:59.265+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:56:59.267+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:59.269+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:59.274+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:59.282+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:59.299+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:56:59.299+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:56:59.299+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:56:59.299+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999ed500df49298418ff17 2014-12-23T16:56:59.334+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:56:59.334+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999ed500df49298418ff17 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-43' 2014-12-23T16:56:59.515+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:59-54999edb9a8f659f34bfb40b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353819515), what: "moveChunk.commit", ns: "benchdb1.COL-43", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:56:59.548+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:59.548+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:56:59.548+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-43 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1272 2014-12-23T16:56:59.548+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-43 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:56:59.548+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:56:59.694+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:56:59.694+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:56:59-54999edb9a8f659f34bfb40c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353819694), what: "moveChunk.from", ns: "benchdb1.COL-43", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 409, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 249, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:56:59.743+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ed500df49298418ff17') } ntoreturn:1 keyUpdates:0 reslen:37 889ms 2014-12-23T16:56:59.745+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-43", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-43-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ed500df49298418ff17') } 2014-12-23T16:57:00.047+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999edb9a8f659f34bfb40d 2014-12-23T16:57:00.047+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-43 based on current shard version 3|0||54999ed500df49298418ff17, current metadata version is 3|0||54999ed500df49298418ff17 2014-12-23T16:57:00.048+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-43 from shard version 3|0||54999ed500df49298418ff17 to shard version 3|1||54999ed500df49298418ff17 2014-12-23T16:57:00.048+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999ed500df49298418ff17, took 0ms 2014-12-23T16:57:00.048+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999ed500df49298418ff17 2014-12-23T16:57:00.220+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:00-54999edc9a8f659f34bfb40e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353820220), what: "split", ns: "benchdb1.COL-43", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ed500df49298418ff17') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ed500df49298418ff17') } } } 2014-12-23T16:57:00.428+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:00.428+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-43", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-43-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ed500df49298418ff17') } ntoreturn:1 keyUpdates:0 reslen:37 682ms 2014-12-23T16:57:02.848+0000 I INDEX [conn2] build index on: benchdb1.COL-44 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-44" } 2014-12-23T16:57:02.848+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:02.849+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:03.182+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-44 with requested shard version 1|2||54999edb00df49298418ff1c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:03.183+0000 I SHARDING [conn2] collection benchdb1.COL-44 was previously unsharded, new metadata loaded with shard version 1|2||54999edb00df49298418ff1c 2014-12-23T16:57:03.183+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999edb00df49298418ff1c, took 0ms 2014-12-23T16:57:03.251+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:03.252+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999edb00df49298418ff1c') } 2014-12-23T16:57:04.140+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999edf9a8f659f34bfb410 2014-12-23T16:57:04.140+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:04-54999ee09a8f659f34bfb411", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353824140), what: "moveChunk.start", ns: "benchdb1.COL-44", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:04.198+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-44 based on current shard version 1|2||54999edb00df49298418ff1c, current metadata version is 1|2||54999edb00df49298418ff1c 2014-12-23T16:57:04.199+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-44 already up to date (shard version : 1|2||54999edb00df49298418ff1c, took 0ms) 2014-12-23T16:57:04.199+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999edb00df49298418ff1c 2014-12-23T16:57:04.199+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:04.202+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:04.204+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:04.209+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:04.209+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:04.209+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:04.209+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999edb00df49298418ff1c 2014-12-23T16:57:04.267+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:04.267+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999edb00df49298418ff1c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-44' 2014-12-23T16:57:04.573+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:04-54999ee09a8f659f34bfb412", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353824573), what: "moveChunk.commit", ns: "benchdb1.COL-44", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:04.634+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:04.634+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:04.634+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-44 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1291 2014-12-23T16:57:04.634+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-44 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:04.634+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:04.937+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:04.937+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:04-54999ee09a8f659f34bfb413", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353824937), what: "moveChunk.from", ns: "benchdb1.COL-44", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 946, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 424, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:05.000+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999edb00df49298418ff1c') } ntoreturn:1 keyUpdates:0 reslen:37 1748ms 2014-12-23T16:57:05.001+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:05.002+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999edb00df49298418ff1c') } 2014-12-23T16:57:05.353+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:57:04 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:57:05.607+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ee19a8f659f34bfb414 2014-12-23T16:57:05.607+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:05-54999ee19a8f659f34bfb415", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353825607), what: "moveChunk.start", ns: "benchdb1.COL-44", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:05.669+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-44 based on current shard version 2|0||54999edb00df49298418ff1c, current metadata version is 2|0||54999edb00df49298418ff1c 2014-12-23T16:57:05.670+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-44 from shard version 2|0||54999edb00df49298418ff1c to shard version 2|1||54999edb00df49298418ff1c 2014-12-23T16:57:05.670+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999edb00df49298418ff1c, took 0ms 2014-12-23T16:57:05.670+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999edb00df49298418ff1c 2014-12-23T16:57:05.670+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:05.672+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:05.675+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:05.679+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:05.688+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:05.704+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:05.704+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:05.704+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:05.705+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999edb00df49298418ff1c 2014-12-23T16:57:05.758+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:05.758+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999edb00df49298418ff1c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-44' 2014-12-23T16:57:06.032+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:06-54999ee29a8f659f34bfb416", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353826032), what: "moveChunk.commit", ns: "benchdb1.COL-44", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:06.101+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:06.101+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:06.101+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-44 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1299 2014-12-23T16:57:06.101+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-44 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:06.101+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:06.404+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:06.404+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:06-54999ee29a8f659f34bfb417", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353826404), what: "moveChunk.from", ns: "benchdb1.COL-44", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 396, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:06.463+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999edb00df49298418ff1c') } ntoreturn:1 keyUpdates:0 reslen:37 1461ms 2014-12-23T16:57:06.465+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-44", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-44-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999edb00df49298418ff1c') } 2014-12-23T16:57:07.072+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ee29a8f659f34bfb418 2014-12-23T16:57:07.072+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-44 based on current shard version 3|0||54999edb00df49298418ff1c, current metadata version is 3|0||54999edb00df49298418ff1c 2014-12-23T16:57:07.073+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-44 from shard version 3|0||54999edb00df49298418ff1c to shard version 3|1||54999edb00df49298418ff1c 2014-12-23T16:57:07.073+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999edb00df49298418ff1c, took 0ms 2014-12-23T16:57:07.073+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999edb00df49298418ff1c 2014-12-23T16:57:07.379+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:07-54999ee39a8f659f34bfb419", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353827379), what: "split", ns: "benchdb1.COL-44", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999edb00df49298418ff1c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999edb00df49298418ff1c') } } } 2014-12-23T16:57:07.774+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:07.774+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-44", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-44-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999edb00df49298418ff1c') } ntoreturn:1 keyUpdates:0 reslen:37 1308ms 2014-12-23T16:57:10.355+0000 I INDEX [conn2] build index on: benchdb1.COL-45 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-45" } 2014-12-23T16:57:10.355+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:10.356+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:10.664+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-45 with requested shard version 1|2||54999ee300df49298418ff1f, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:10.665+0000 I SHARDING [conn2] collection benchdb1.COL-45 was previously unsharded, new metadata loaded with shard version 1|2||54999ee300df49298418ff1f 2014-12-23T16:57:10.665+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ee300df49298418ff1f, took 0ms 2014-12-23T16:57:10.706+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:10.706+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee300df49298418ff1f') } 2014-12-23T16:57:11.472+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ee69a8f659f34bfb41b 2014-12-23T16:57:11.472+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:11-54999ee79a8f659f34bfb41c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353831472), what: "moveChunk.start", ns: "benchdb1.COL-45", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:11.520+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-45 based on current shard version 1|2||54999ee300df49298418ff1f, current metadata version is 1|2||54999ee300df49298418ff1f 2014-12-23T16:57:11.521+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-45 already up to date (shard version : 1|2||54999ee300df49298418ff1f, took 0ms) 2014-12-23T16:57:11.521+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999ee300df49298418ff1f 2014-12-23T16:57:11.522+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:11.524+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:11.527+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:11.531+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:11.531+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:11.531+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:11.532+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999ee300df49298418ff1f 2014-12-23T16:57:11.582+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:11.583+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999ee300df49298418ff1f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-45' 2014-12-23T16:57:11.795+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:11-54999ee79a8f659f34bfb41d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353831795), what: "moveChunk.commit", ns: "benchdb1.COL-45", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:11.847+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:11.847+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:11.847+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-45 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1318 2014-12-23T16:57:11.847+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-45 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:11.847+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:12.017+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:12.017+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:12-54999ee89a8f659f34bfb41e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353832017), what: "moveChunk.from", ns: "benchdb1.COL-45", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 815, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 315, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:12.070+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee300df49298418ff1f') } ntoreturn:1 keyUpdates:0 reslen:37 1364ms 2014-12-23T16:57:12.071+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:12.072+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee300df49298418ff1f') } 2014-12-23T16:57:12.382+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ee89a8f659f34bfb41f 2014-12-23T16:57:12.382+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:12-54999ee89a8f659f34bfb420", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353832382), what: "moveChunk.start", ns: "benchdb1.COL-45", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:12.453+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-45 based on current shard version 2|0||54999ee300df49298418ff1f, current metadata version is 2|0||54999ee300df49298418ff1f 2014-12-23T16:57:12.454+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-45 from shard version 2|0||54999ee300df49298418ff1f to shard version 2|1||54999ee300df49298418ff1f 2014-12-23T16:57:12.454+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999ee300df49298418ff1f, took 0ms 2014-12-23T16:57:12.454+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999ee300df49298418ff1f 2014-12-23T16:57:12.454+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:12.457+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:12.459+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:12.464+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:12.472+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:12.489+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:12.489+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:12.489+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:12.489+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999ee300df49298418ff1f 2014-12-23T16:57:12.557+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:12.557+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999ee300df49298418ff1f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-45' 2014-12-23T16:57:12.754+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:12-54999ee89a8f659f34bfb421", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353832754), what: "moveChunk.commit", ns: "benchdb1.COL-45", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:12.810+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:12.810+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:12.810+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-45 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1328 2014-12-23T16:57:12.810+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-45 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:12.810+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:12.969+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:12.969+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:12-54999ee89a8f659f34bfb422", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353832969), what: "moveChunk.from", ns: "benchdb1.COL-45", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 382, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 321, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:13.030+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee300df49298418ff1f') } ntoreturn:1 keyUpdates:0 reslen:37 959ms 2014-12-23T16:57:13.032+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-45", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-45-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ee300df49298418ff1f') } 2014-12-23T16:57:13.382+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ee99a8f659f34bfb423 2014-12-23T16:57:13.383+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-45 based on current shard version 3|0||54999ee300df49298418ff1f, current metadata version is 3|0||54999ee300df49298418ff1f 2014-12-23T16:57:13.383+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-45 from shard version 3|0||54999ee300df49298418ff1f to shard version 3|1||54999ee300df49298418ff1f 2014-12-23T16:57:13.383+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999ee300df49298418ff1f, took 0ms 2014-12-23T16:57:13.384+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999ee300df49298418ff1f 2014-12-23T16:57:13.688+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:13-54999ee99a8f659f34bfb424", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353833688), what: "split", ns: "benchdb1.COL-45", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ee300df49298418ff1f') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ee300df49298418ff1f') } } } 2014-12-23T16:57:14.070+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:14.070+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-45", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-45-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ee300df49298418ff1f') } ntoreturn:1 keyUpdates:0 reslen:37 1038ms 2014-12-23T16:57:16.705+0000 I INDEX [conn2] build index on: benchdb1.COL-46 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-46" } 2014-12-23T16:57:16.705+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:16.706+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:17.015+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-46 with requested shard version 1|2||54999ee900df49298418ff24, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:17.015+0000 I SHARDING [conn2] collection benchdb1.COL-46 was previously unsharded, new metadata loaded with shard version 1|2||54999ee900df49298418ff24 2014-12-23T16:57:17.015+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ee900df49298418ff24, took 0ms 2014-12-23T16:57:17.060+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:17.060+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee900df49298418ff24') } 2014-12-23T16:57:17.970+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eed9a8f659f34bfb426 2014-12-23T16:57:17.970+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:17-54999eed9a8f659f34bfb427", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353837970), what: "moveChunk.start", ns: "benchdb1.COL-46", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:18.031+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-46 based on current shard version 1|2||54999ee900df49298418ff24, current metadata version is 1|2||54999ee900df49298418ff24 2014-12-23T16:57:18.032+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-46 already up to date (shard version : 1|2||54999ee900df49298418ff24, took 0ms) 2014-12-23T16:57:18.032+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999ee900df49298418ff24 2014-12-23T16:57:18.033+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:18.036+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:18.038+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:18.042+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:18.042+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:18.043+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:18.043+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999ee900df49298418ff24 2014-12-23T16:57:18.095+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:18.095+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999ee900df49298418ff24 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-46' 2014-12-23T16:57:18.400+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:18-54999eee9a8f659f34bfb428", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353838400), what: "moveChunk.commit", ns: "benchdb1.COL-46", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:18.459+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:18.459+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:18.459+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-46 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1346 2014-12-23T16:57:18.459+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-46 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:18.459+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:18.762+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:18.762+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:18-54999eee9a8f659f34bfb429", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353838762), what: "moveChunk.from", ns: "benchdb1.COL-46", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 972, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 416, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:18.816+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee900df49298418ff24') } ntoreturn:1 keyUpdates:0 reslen:37 1756ms 2014-12-23T16:57:18.818+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:18.818+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee900df49298418ff24') } 2014-12-23T16:57:19.425+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999eee9a8f659f34bfb42a 2014-12-23T16:57:19.425+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:19-54999eef9a8f659f34bfb42b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353839425), what: "moveChunk.start", ns: "benchdb1.COL-46", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:19.478+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-46 based on current shard version 2|0||54999ee900df49298418ff24, current metadata version is 2|0||54999ee900df49298418ff24 2014-12-23T16:57:19.479+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-46 from shard version 2|0||54999ee900df49298418ff24 to shard version 2|1||54999ee900df49298418ff24 2014-12-23T16:57:19.479+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999ee900df49298418ff24, took 0ms 2014-12-23T16:57:19.479+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999ee900df49298418ff24 2014-12-23T16:57:19.479+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:19.482+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:19.484+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:19.488+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:19.497+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:19.514+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:19.514+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:19.514+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:19.514+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999ee900df49298418ff24 2014-12-23T16:57:19.569+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:19.569+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999ee900df49298418ff24 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-46' 2014-12-23T16:57:19.874+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:19-54999eef9a8f659f34bfb42c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353839874), what: "moveChunk.commit", ns: "benchdb1.COL-46", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:19.937+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:19.937+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:19.937+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-46 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1354 2014-12-23T16:57:19.937+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-46 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:19.937+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:20.240+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:20.240+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:20-54999ef09a8f659f34bfb42d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353840240), what: "moveChunk.from", ns: "benchdb1.COL-46", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 661, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:20.293+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ee900df49298418ff24') } ntoreturn:1 keyUpdates:0 reslen:37 1475ms 2014-12-23T16:57:20.295+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-46", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-46-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ee900df49298418ff24') } 2014-12-23T16:57:20.852+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ef09a8f659f34bfb42e 2014-12-23T16:57:20.853+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-46 based on current shard version 3|0||54999ee900df49298418ff24, current metadata version is 3|0||54999ee900df49298418ff24 2014-12-23T16:57:20.853+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-46 from shard version 3|0||54999ee900df49298418ff24 to shard version 3|1||54999ee900df49298418ff24 2014-12-23T16:57:20.853+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999ee900df49298418ff24, took 0ms 2014-12-23T16:57:20.853+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999ee900df49298418ff24 2014-12-23T16:57:21.158+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:21-54999ef19a8f659f34bfb42f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353841158), what: "split", ns: "benchdb1.COL-46", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ee900df49298418ff24') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ee900df49298418ff24') } } } 2014-12-23T16:57:21.545+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:21.545+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-46", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-46-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ee900df49298418ff24') } ntoreturn:1 keyUpdates:0 reslen:37 1250ms 2014-12-23T16:57:24.072+0000 I INDEX [conn2] build index on: benchdb1.COL-47 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-47" } 2014-12-23T16:57:24.072+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:24.073+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:24.382+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-47 with requested shard version 1|2||54999ef100df49298418ff28, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:24.383+0000 I SHARDING [conn2] collection benchdb1.COL-47 was previously unsharded, new metadata loaded with shard version 1|2||54999ef100df49298418ff28 2014-12-23T16:57:24.383+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ef100df49298418ff28, took 0ms 2014-12-23T16:57:24.427+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:24.427+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef100df49298418ff28') } 2014-12-23T16:57:24.891+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ef49a8f659f34bfb431 2014-12-23T16:57:24.891+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:24-54999ef49a8f659f34bfb432", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353844891), what: "moveChunk.start", ns: "benchdb1.COL-47", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:24.943+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-47 based on current shard version 1|2||54999ef100df49298418ff28, current metadata version is 1|2||54999ef100df49298418ff28 2014-12-23T16:57:24.943+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-47 already up to date (shard version : 1|2||54999ef100df49298418ff28, took 0ms) 2014-12-23T16:57:24.943+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999ef100df49298418ff28 2014-12-23T16:57:24.944+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:24.947+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:24.949+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:24.953+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:24.953+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:24.954+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:24.954+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999ef100df49298418ff28 2014-12-23T16:57:25.019+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:25.019+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999ef100df49298418ff28 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-47' 2014-12-23T16:57:25.187+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:25-54999ef59a8f659f34bfb433", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353845187), what: "moveChunk.commit", ns: "benchdb1.COL-47", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:25.241+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:25.241+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:25.241+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-47 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1375 2014-12-23T16:57:25.241+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-47 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:25.241+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:25.413+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:25.413+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:25-54999ef59a8f659f34bfb434", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353845413), what: "moveChunk.from", ns: "benchdb1.COL-47", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 516, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 287, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:25.440+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef100df49298418ff28') } ntoreturn:1 keyUpdates:0 reslen:37 1013ms 2014-12-23T16:57:25.442+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:25.442+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef100df49298418ff28') } 2014-12-23T16:57:25.738+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ef59a8f659f34bfb435 2014-12-23T16:57:25.738+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:25-54999ef59a8f659f34bfb436", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353845738), what: "moveChunk.start", ns: "benchdb1.COL-47", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:25.792+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-47 based on current shard version 2|0||54999ef100df49298418ff28, current metadata version is 2|0||54999ef100df49298418ff28 2014-12-23T16:57:25.793+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-47 from shard version 2|0||54999ef100df49298418ff28 to shard version 2|1||54999ef100df49298418ff28 2014-12-23T16:57:25.793+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999ef100df49298418ff28, took 0ms 2014-12-23T16:57:25.793+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999ef100df49298418ff28 2014-12-23T16:57:25.793+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:25.795+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:25.798+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:25.802+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:25.811+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:25.827+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:25.827+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:25.827+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:25.827+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999ef100df49298418ff28 2014-12-23T16:57:25.887+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:25.887+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999ef100df49298418ff28 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-47' 2014-12-23T16:57:26.061+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:26-54999ef69a8f659f34bfb437", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353846061), what: "moveChunk.commit", ns: "benchdb1.COL-47", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:26.114+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:26.114+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:26.114+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-47 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1383 2014-12-23T16:57:26.114+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-47 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:26.114+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:26.356+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:26.356+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:26-54999ef69a8f659f34bfb438", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353846356), what: "moveChunk.from", ns: "benchdb1.COL-47", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 350, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 286, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:26.411+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef100df49298418ff28') } ntoreturn:1 keyUpdates:0 reslen:37 968ms 2014-12-23T16:57:26.412+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-47", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-47-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ef100df49298418ff28') } 2014-12-23T16:57:27.020+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999ef69a8f659f34bfb439 2014-12-23T16:57:27.020+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-47 based on current shard version 3|0||54999ef100df49298418ff28, current metadata version is 3|0||54999ef100df49298418ff28 2014-12-23T16:57:27.020+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-47 from shard version 3|0||54999ef100df49298418ff28 to shard version 3|1||54999ef100df49298418ff28 2014-12-23T16:57:27.020+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999ef100df49298418ff28, took 0ms 2014-12-23T16:57:27.020+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999ef100df49298418ff28 2014-12-23T16:57:27.326+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:27-54999ef79a8f659f34bfb43a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353847325), what: "split", ns: "benchdb1.COL-47", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ef100df49298418ff28') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ef100df49298418ff28') } } } 2014-12-23T16:57:27.714+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:27.714+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-47", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-47-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ef100df49298418ff28') } ntoreturn:1 keyUpdates:0 reslen:37 1301ms 2014-12-23T16:57:30.336+0000 I INDEX [conn2] build index on: benchdb1.COL-48 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-48" } 2014-12-23T16:57:30.336+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:30.337+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:30.652+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-48 with requested shard version 1|2||54999ef700df49298418ff2c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:30.652+0000 I SHARDING [conn2] collection benchdb1.COL-48 was previously unsharded, new metadata loaded with shard version 1|2||54999ef700df49298418ff2c 2014-12-23T16:57:30.652+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999ef700df49298418ff2c, took 0ms 2014-12-23T16:57:30.736+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:30.736+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef700df49298418ff2c') } 2014-12-23T16:57:31.645+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999efb9a8f659f34bfb43c 2014-12-23T16:57:31.645+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:31-54999efb9a8f659f34bfb43d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353851645), what: "moveChunk.start", ns: "benchdb1.COL-48", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:31.714+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-48 based on current shard version 1|2||54999ef700df49298418ff2c, current metadata version is 1|2||54999ef700df49298418ff2c 2014-12-23T16:57:31.715+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-48 already up to date (shard version : 1|2||54999ef700df49298418ff2c, took 0ms) 2014-12-23T16:57:31.715+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999ef700df49298418ff2c 2014-12-23T16:57:31.716+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:31.718+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:31.721+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:31.725+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:31.725+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:31.726+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:31.726+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999ef700df49298418ff2c 2014-12-23T16:57:31.775+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:31.775+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999ef700df49298418ff2c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-48' 2014-12-23T16:57:32.080+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:32-54999efc9a8f659f34bfb43e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353852080), what: "moveChunk.commit", ns: "benchdb1.COL-48", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:32.136+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:32.137+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:32.137+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-48 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1402 2014-12-23T16:57:32.137+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-48 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:32.137+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:32.440+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:32.440+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:32-54999efc9a8f659f34bfb43f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353852440), what: "moveChunk.from", ns: "benchdb1.COL-48", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 979, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 411, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:32.499+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef700df49298418ff2c') } ntoreturn:1 keyUpdates:0 reslen:37 1762ms 2014-12-23T16:57:32.500+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:32.501+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef700df49298418ff2c') } 2014-12-23T16:57:33.107+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999efc9a8f659f34bfb440 2014-12-23T16:57:33.107+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:33-54999efd9a8f659f34bfb441", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353853107), what: "moveChunk.start", ns: "benchdb1.COL-48", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:33.164+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-48 based on current shard version 2|0||54999ef700df49298418ff2c, current metadata version is 2|0||54999ef700df49298418ff2c 2014-12-23T16:57:33.165+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-48 from shard version 2|0||54999ef700df49298418ff2c to shard version 2|1||54999ef700df49298418ff2c 2014-12-23T16:57:33.165+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999ef700df49298418ff2c, took 0ms 2014-12-23T16:57:33.165+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999ef700df49298418ff2c 2014-12-23T16:57:33.165+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:33.167+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:33.170+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:33.174+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:33.182+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:33.199+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:33.199+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:33.199+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:33.199+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999ef700df49298418ff2c 2014-12-23T16:57:33.246+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:33.247+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999ef700df49298418ff2c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-48' 2014-12-23T16:57:33.535+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:33-54999efd9a8f659f34bfb442", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353853535), what: "moveChunk.commit", ns: "benchdb1.COL-48", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:33.589+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:33.589+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:33.589+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-48 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1410 2014-12-23T16:57:33.589+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-48 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:33.589+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:33.892+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:33.892+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:33-54999efd9a8f659f34bfb443", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353853892), what: "moveChunk.from", ns: "benchdb1.COL-48", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 664, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 390, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:33.948+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999ef700df49298418ff2c') } ntoreturn:1 keyUpdates:0 reslen:37 1447ms 2014-12-23T16:57:33.950+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-48", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-48-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ef700df49298418ff2c') } 2014-12-23T16:57:34.556+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999efd9a8f659f34bfb444 2014-12-23T16:57:34.556+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-48 based on current shard version 3|0||54999ef700df49298418ff2c, current metadata version is 3|0||54999ef700df49298418ff2c 2014-12-23T16:57:34.557+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-48 from shard version 3|0||54999ef700df49298418ff2c to shard version 3|1||54999ef700df49298418ff2c 2014-12-23T16:57:34.557+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999ef700df49298418ff2c, took 0ms 2014-12-23T16:57:34.557+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999ef700df49298418ff2c 2014-12-23T16:57:34.861+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:34-54999efe9a8f659f34bfb445", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353854861), what: "split", ns: "benchdb1.COL-48", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999ef700df49298418ff2c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999ef700df49298418ff2c') } } } 2014-12-23T16:57:35.244+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:35.244+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-48", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-48-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999ef700df49298418ff2c') } ntoreturn:1 keyUpdates:0 reslen:37 1294ms 2014-12-23T16:57:35.856+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:57:35 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:57:37.383+0000 I INDEX [conn2] build index on: benchdb1.COL-49 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-49" } 2014-12-23T16:57:37.383+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:37.384+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:37.693+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-49 with requested shard version 1|2||54999efe00df49298418ff30, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:37.694+0000 I SHARDING [conn2] collection benchdb1.COL-49 was previously unsharded, new metadata loaded with shard version 1|2||54999efe00df49298418ff30 2014-12-23T16:57:37.694+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999efe00df49298418ff30, took 0ms 2014-12-23T16:57:37.755+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:37.756+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999efe00df49298418ff30') } 2014-12-23T16:57:38.120+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f019a8f659f34bfb447 2014-12-23T16:57:38.120+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:38-54999f029a8f659f34bfb448", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353858120), what: "moveChunk.start", ns: "benchdb1.COL-49", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:38.219+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-49 based on current shard version 1|2||54999efe00df49298418ff30, current metadata version is 1|2||54999efe00df49298418ff30 2014-12-23T16:57:38.220+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-49 already up to date (shard version : 1|2||54999efe00df49298418ff30, took 0ms) 2014-12-23T16:57:38.220+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999efe00df49298418ff30 2014-12-23T16:57:38.220+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:38.223+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:38.225+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:38.230+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:38.230+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:38.230+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:38.230+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999efe00df49298418ff30 2014-12-23T16:57:38.277+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:38.277+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999efe00df49298418ff30 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-49' 2014-12-23T16:57:38.505+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:38-54999f029a8f659f34bfb449", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353858505), what: "moveChunk.commit", ns: "benchdb1.COL-49", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:38.557+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:38.557+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:38.557+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-49 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1431 2014-12-23T16:57:38.557+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-49 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:38.557+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:38.715+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:38.715+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:38-54999f029a8f659f34bfb44a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353858715), what: "moveChunk.from", ns: "benchdb1.COL-49", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 464, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 327, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:38.769+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999efe00df49298418ff30') } ntoreturn:1 keyUpdates:0 reslen:37 1014ms 2014-12-23T16:57:38.771+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:38.771+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999efe00df49298418ff30') } 2014-12-23T16:57:39.062+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f029a8f659f34bfb44b 2014-12-23T16:57:39.063+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:39-54999f039a8f659f34bfb44c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353859063), what: "moveChunk.start", ns: "benchdb1.COL-49", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:39.114+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-49 based on current shard version 2|0||54999efe00df49298418ff30, current metadata version is 2|0||54999efe00df49298418ff30 2014-12-23T16:57:39.115+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-49 from shard version 2|0||54999efe00df49298418ff30 to shard version 2|1||54999efe00df49298418ff30 2014-12-23T16:57:39.115+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999efe00df49298418ff30, took 0ms 2014-12-23T16:57:39.115+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999efe00df49298418ff30 2014-12-23T16:57:39.115+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:39.117+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:39.120+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:39.124+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:39.132+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:39.149+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:39.149+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:39.149+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:39.149+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999efe00df49298418ff30 2014-12-23T16:57:39.179+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:39.179+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999efe00df49298418ff30 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-49' 2014-12-23T16:57:39.365+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:39-54999f039a8f659f34bfb44d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353859365), what: "moveChunk.commit", ns: "benchdb1.COL-49", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:39.410+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:39.410+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:39.410+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-49 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1439 2014-12-23T16:57:39.410+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-49 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:39.410+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:39.614+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:39.614+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:39-54999f039a8f659f34bfb44e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353859614), what: "moveChunk.from", ns: "benchdb1.COL-49", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 343, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 260, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:39.670+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999efe00df49298418ff30') } ntoreturn:1 keyUpdates:0 reslen:37 899ms 2014-12-23T16:57:39.672+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-49", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-49-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999efe00df49298418ff30') } 2014-12-23T16:57:40.261+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f039a8f659f34bfb44f 2014-12-23T16:57:40.261+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-49 based on current shard version 3|0||54999efe00df49298418ff30, current metadata version is 3|0||54999efe00df49298418ff30 2014-12-23T16:57:40.262+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-49 from shard version 3|0||54999efe00df49298418ff30 to shard version 3|1||54999efe00df49298418ff30 2014-12-23T16:57:40.262+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999efe00df49298418ff30, took 0ms 2014-12-23T16:57:40.262+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999efe00df49298418ff30 2014-12-23T16:57:40.567+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:40-54999f049a8f659f34bfb450", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353860566), what: "split", ns: "benchdb1.COL-49", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999efe00df49298418ff30') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999efe00df49298418ff30') } } } 2014-12-23T16:57:40.946+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:40.946+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-49", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-49-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999efe00df49298418ff30') } ntoreturn:1 keyUpdates:0 reslen:37 1273ms 2014-12-23T16:57:43.494+0000 I INDEX [conn2] build index on: benchdb1.COL-50 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-50" } 2014-12-23T16:57:43.494+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:43.495+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:43.810+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-50 with requested shard version 1|2||54999f0400df49298418ff34, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:43.811+0000 I SHARDING [conn2] collection benchdb1.COL-50 was previously unsharded, new metadata loaded with shard version 1|2||54999f0400df49298418ff34 2014-12-23T16:57:43.811+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f0400df49298418ff34, took 0ms 2014-12-23T16:57:43.855+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:43.855+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0400df49298418ff34') } 2014-12-23T16:57:44.765+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f089a8f659f34bfb452 2014-12-23T16:57:44.765+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:44-54999f089a8f659f34bfb453", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353864765), what: "moveChunk.start", ns: "benchdb1.COL-50", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:44.831+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-50 based on current shard version 1|2||54999f0400df49298418ff34, current metadata version is 1|2||54999f0400df49298418ff34 2014-12-23T16:57:44.832+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-50 already up to date (shard version : 1|2||54999f0400df49298418ff34, took 0ms) 2014-12-23T16:57:44.832+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999f0400df49298418ff34 2014-12-23T16:57:44.833+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:44.836+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:44.838+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:44.842+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:44.842+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:44.843+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:44.843+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999f0400df49298418ff34 2014-12-23T16:57:44.898+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:44.898+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999f0400df49298418ff34 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-50' 2014-12-23T16:57:45.202+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:45-54999f099a8f659f34bfb454", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353865202), what: "moveChunk.commit", ns: "benchdb1.COL-50", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:45.261+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:45.261+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:45.261+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-50 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1457 2014-12-23T16:57:45.261+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-50 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:45.261+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:45.564+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:45.564+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:45-54999f099a8f659f34bfb455", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353865564), what: "moveChunk.from", ns: "benchdb1.COL-50", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 977, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 418, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:45.630+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0400df49298418ff34') } ntoreturn:1 keyUpdates:0 reslen:37 1775ms 2014-12-23T16:57:45.631+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:45.631+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0400df49298418ff34') } 2014-12-23T16:57:46.238+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f099a8f659f34bfb456 2014-12-23T16:57:46.238+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:46-54999f0a9a8f659f34bfb457", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353866238), what: "moveChunk.start", ns: "benchdb1.COL-50", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:46.294+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-50 based on current shard version 2|0||54999f0400df49298418ff34, current metadata version is 2|0||54999f0400df49298418ff34 2014-12-23T16:57:46.295+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-50 from shard version 2|0||54999f0400df49298418ff34 to shard version 2|1||54999f0400df49298418ff34 2014-12-23T16:57:46.295+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999f0400df49298418ff34, took 0ms 2014-12-23T16:57:46.295+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999f0400df49298418ff34 2014-12-23T16:57:46.295+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:46.297+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:46.300+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:46.304+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:46.312+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:46.329+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:46.329+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:46.329+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:46.329+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999f0400df49298418ff34 2014-12-23T16:57:46.376+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:46.376+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999f0400df49298418ff34 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-50' 2014-12-23T16:57:46.666+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:46-54999f0a9a8f659f34bfb458", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353866666), what: "moveChunk.commit", ns: "benchdb1.COL-50", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:46.739+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:46.739+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:46.739+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-50 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1465 2014-12-23T16:57:46.740+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-50 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:46.740+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:47.043+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:47.043+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:47-54999f0b9a8f659f34bfb459", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353867043), what: "moveChunk.from", ns: "benchdb1.COL-50", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 410, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:47.092+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0400df49298418ff34') } ntoreturn:1 keyUpdates:0 reslen:37 1461ms 2014-12-23T16:57:47.094+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-50", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-50-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f0400df49298418ff34') } 2014-12-23T16:57:47.701+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f0b9a8f659f34bfb45a 2014-12-23T16:57:47.701+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-50 based on current shard version 3|0||54999f0400df49298418ff34, current metadata version is 3|0||54999f0400df49298418ff34 2014-12-23T16:57:47.701+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-50 from shard version 3|0||54999f0400df49298418ff34 to shard version 3|1||54999f0400df49298418ff34 2014-12-23T16:57:47.702+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999f0400df49298418ff34, took 0ms 2014-12-23T16:57:47.702+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999f0400df49298418ff34 2014-12-23T16:57:48.006+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:48-54999f0c9a8f659f34bfb45b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353868006), what: "split", ns: "benchdb1.COL-50", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f0400df49298418ff34') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f0400df49298418ff34') } } } 2014-12-23T16:57:48.400+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:48.400+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-50", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-50-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f0400df49298418ff34') } ntoreturn:1 keyUpdates:0 reslen:37 1306ms 2014-12-23T16:57:50.511+0000 I INDEX [conn2] build index on: benchdb1.COL-51 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-51" } 2014-12-23T16:57:50.511+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:50.512+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:50.810+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-51 with requested shard version 1|2||54999f0b00df49298418ff38, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:50.810+0000 I SHARDING [conn2] collection benchdb1.COL-51 was previously unsharded, new metadata loaded with shard version 1|2||54999f0b00df49298418ff38 2014-12-23T16:57:50.810+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f0b00df49298418ff38, took 0ms 2014-12-23T16:57:50.855+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:50.856+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0b00df49298418ff38') } 2014-12-23T16:57:51.245+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f0e9a8f659f34bfb45d 2014-12-23T16:57:51.245+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:51-54999f0f9a8f659f34bfb45e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353871245), what: "moveChunk.start", ns: "benchdb1.COL-51", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:51.297+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-51 based on current shard version 1|2||54999f0b00df49298418ff38, current metadata version is 1|2||54999f0b00df49298418ff38 2014-12-23T16:57:51.297+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-51 already up to date (shard version : 1|2||54999f0b00df49298418ff38, took 0ms) 2014-12-23T16:57:51.297+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999f0b00df49298418ff38 2014-12-23T16:57:51.298+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:51.300+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:51.303+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:51.307+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:51.307+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:51.308+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:51.308+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999f0b00df49298418ff38 2014-12-23T16:57:51.351+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:51.352+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999f0b00df49298418ff38 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-51' 2014-12-23T16:57:51.556+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:51-54999f0f9a8f659f34bfb45f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353871556), what: "moveChunk.commit", ns: "benchdb1.COL-51", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:51.613+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:51.613+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:51.613+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-51 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1485 2014-12-23T16:57:51.613+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-51 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:51.613+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:51.797+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:51.797+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:51-54999f0f9a8f659f34bfb460", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353871797), what: "moveChunk.from", ns: "benchdb1.COL-51", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 441, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 305, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:51.849+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0b00df49298418ff38') } ntoreturn:1 keyUpdates:0 reslen:37 993ms 2014-12-23T16:57:51.850+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:51.851+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0b00df49298418ff38') } 2014-12-23T16:57:52.121+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f0f9a8f659f34bfb461 2014-12-23T16:57:52.121+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:52-54999f109a8f659f34bfb462", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353872121), what: "moveChunk.start", ns: "benchdb1.COL-51", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:52.173+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-51 based on current shard version 2|0||54999f0b00df49298418ff38, current metadata version is 2|0||54999f0b00df49298418ff38 2014-12-23T16:57:52.174+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-51 from shard version 2|0||54999f0b00df49298418ff38 to shard version 2|1||54999f0b00df49298418ff38 2014-12-23T16:57:52.174+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999f0b00df49298418ff38, took 0ms 2014-12-23T16:57:52.174+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999f0b00df49298418ff38 2014-12-23T16:57:52.174+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:52.177+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:52.179+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:52.183+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:52.192+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:52.208+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:52.208+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:52.209+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:52.209+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999f0b00df49298418ff38 2014-12-23T16:57:52.310+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:52.310+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999f0b00df49298418ff38 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-51' 2014-12-23T16:57:52.544+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:52-54999f109a8f659f34bfb463", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353872544), what: "moveChunk.commit", ns: "benchdb1.COL-51", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:52.617+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:52.617+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:52.617+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-51 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1493 2014-12-23T16:57:52.617+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-51 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:57:52.617+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:52.920+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:52.920+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:52-54999f109a8f659f34bfb464", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353872920), what: "moveChunk.from", ns: "benchdb1.COL-51", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 323, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 408, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:57:52.980+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f0b00df49298418ff38') } ntoreturn:1 keyUpdates:0 reslen:37 1129ms 2014-12-23T16:57:52.982+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-51", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-51-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f0b00df49298418ff38') } 2014-12-23T16:57:53.588+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f109a8f659f34bfb465 2014-12-23T16:57:53.588+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-51 based on current shard version 3|0||54999f0b00df49298418ff38, current metadata version is 3|0||54999f0b00df49298418ff38 2014-12-23T16:57:53.589+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-51 from shard version 3|0||54999f0b00df49298418ff38 to shard version 3|1||54999f0b00df49298418ff38 2014-12-23T16:57:53.589+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999f0b00df49298418ff38, took 0ms 2014-12-23T16:57:53.589+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999f0b00df49298418ff38 2014-12-23T16:57:53.893+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:53-54999f119a8f659f34bfb466", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353873893), what: "split", ns: "benchdb1.COL-51", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f0b00df49298418ff38') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f0b00df49298418ff38') } } } 2014-12-23T16:57:54.279+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:54.279+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-51", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-51-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f0b00df49298418ff38') } ntoreturn:1 keyUpdates:0 reslen:37 1297ms 2014-12-23T16:57:56.881+0000 I INDEX [conn2] build index on: benchdb1.COL-52 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-52" } 2014-12-23T16:57:56.881+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:57:56.882+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:57:57.184+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-52 with requested shard version 1|2||54999f1100df49298418ff3c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:57:57.185+0000 I SHARDING [conn2] collection benchdb1.COL-52 was previously unsharded, new metadata loaded with shard version 1|2||54999f1100df49298418ff3c 2014-12-23T16:57:57.185+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f1100df49298418ff3c, took 0ms 2014-12-23T16:57:57.224+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:57.224+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1100df49298418ff3c') } 2014-12-23T16:57:58.133+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f159a8f659f34bfb468 2014-12-23T16:57:58.133+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:58-54999f169a8f659f34bfb469", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353878133), what: "moveChunk.start", ns: "benchdb1.COL-52", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:57:58.199+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 based on current shard version 1|2||54999f1100df49298418ff3c, current metadata version is 1|2||54999f1100df49298418ff3c 2014-12-23T16:57:58.199+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-52 already up to date (shard version : 1|2||54999f1100df49298418ff3c, took 0ms) 2014-12-23T16:57:58.199+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999f1100df49298418ff3c 2014-12-23T16:57:58.200+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:58.203+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:58.205+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:58.209+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:58.209+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:58.210+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:58.210+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999f1100df49298418ff3c 2014-12-23T16:57:58.260+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:58.260+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999f1100df49298418ff3c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-52' 2014-12-23T16:57:58.565+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:58-54999f169a8f659f34bfb46a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353878565), what: "moveChunk.commit", ns: "benchdb1.COL-52", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:57:58.626+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:58.626+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:57:58.626+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-52 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1512 2014-12-23T16:57:58.626+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-52 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:57:58.626+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:57:58.929+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:57:58.929+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:58-54999f169a8f659f34bfb46b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353878929), what: "moveChunk.from", ns: "benchdb1.COL-52", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 975, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 416, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:57:58.987+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1100df49298418ff3c') } ntoreturn:1 keyUpdates:0 reslen:37 1763ms 2014-12-23T16:57:58.989+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:57:58.989+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1100df49298418ff3c') } 2014-12-23T16:57:59.520+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f169a8f659f34bfb46c 2014-12-23T16:57:59.521+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:59-54999f179a8f659f34bfb46d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353879521), what: "moveChunk.start", ns: "benchdb1.COL-52", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:57:59.577+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 based on current shard version 2|0||54999f1100df49298418ff3c, current metadata version is 2|0||54999f1100df49298418ff3c 2014-12-23T16:57:59.577+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-52 from shard version 2|0||54999f1100df49298418ff3c to shard version 2|1||54999f1100df49298418ff3c 2014-12-23T16:57:59.577+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999f1100df49298418ff3c, took 0ms 2014-12-23T16:57:59.577+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999f1100df49298418ff3c 2014-12-23T16:57:59.577+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:57:59.580+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:59.582+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:59.587+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:59.595+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:59.611+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:57:59.611+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:57:59.612+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:57:59.612+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999f1100df49298418ff3c 2014-12-23T16:57:59.665+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:57:59.665+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999f1100df49298418ff3c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-52' 2014-12-23T16:57:59.962+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:57:59-54999f179a8f659f34bfb46e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353879962), what: "moveChunk.commit", ns: "benchdb1.COL-52", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:00.018+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:00.018+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:58:00.018+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-52 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1520 2014-12-23T16:58:00.019+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-52 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:00.019+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:00.321+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:00.321+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:00-54999f189a8f659f34bfb46f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353880321), what: "moveChunk.from", ns: "benchdb1.COL-52", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 588, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 406, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:00.379+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1100df49298418ff3c') } ntoreturn:1 keyUpdates:0 reslen:37 1390ms 2014-12-23T16:58:00.381+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-52", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-52-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f1100df49298418ff3c') } 2014-12-23T16:58:00.987+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f189a8f659f34bfb470 2014-12-23T16:58:00.987+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 based on current shard version 3|0||54999f1100df49298418ff3c, current metadata version is 3|0||54999f1100df49298418ff3c 2014-12-23T16:58:00.988+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-52 from shard version 3|0||54999f1100df49298418ff3c to shard version 3|1||54999f1100df49298418ff3c 2014-12-23T16:58:00.988+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999f1100df49298418ff3c, took 0ms 2014-12-23T16:58:00.988+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999f1100df49298418ff3c 2014-12-23T16:58:01.292+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:01-54999f199a8f659f34bfb471", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353881292), what: "split", ns: "benchdb1.COL-52", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f1100df49298418ff3c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f1100df49298418ff3c') } } } 2014-12-23T16:58:01.712+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:01.712+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-52", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-52-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f1100df49298418ff3c') } ntoreturn:1 keyUpdates:0 reslen:37 1331ms 2014-12-23T16:58:03.791+0000 I INDEX [conn2] build index on: benchdb1.COL-53 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-53" } 2014-12-23T16:58:03.791+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:03.793+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:04.140+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-53 with requested shard version 1|2||54999f1800df49298418ff40, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:04.150+0000 I SHARDING [conn2] collection benchdb1.COL-53 was previously unsharded, new metadata loaded with shard version 1|2||54999f1800df49298418ff40 2014-12-23T16:58:04.150+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f1800df49298418ff40, took 10ms 2014-12-23T16:58:04.218+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:04.218+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1800df49298418ff40') } 2014-12-23T16:58:04.620+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f1c9a8f659f34bfb473 2014-12-23T16:58:04.620+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:04-54999f1c9a8f659f34bfb474", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353884620), what: "moveChunk.start", ns: "benchdb1.COL-53", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:04.673+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-53 based on current shard version 1|2||54999f1800df49298418ff40, current metadata version is 1|2||54999f1800df49298418ff40 2014-12-23T16:58:04.673+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-53 already up to date (shard version : 1|2||54999f1800df49298418ff40, took 0ms) 2014-12-23T16:58:04.674+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f1800df49298418ff40 2014-12-23T16:58:04.674+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:04.677+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:04.679+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:04.688+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:04.696+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:04.696+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:04.697+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:04.697+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f1800df49298418ff40 2014-12-23T16:58:04.731+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:04.732+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f1800df49298418ff40 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-53' 2014-12-23T16:58:04.907+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:04-54999f1c9a8f659f34bfb475", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353884907), what: "moveChunk.commit", ns: "benchdb1.COL-53", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:04.964+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:04.964+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:04.964+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-53 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1541 2014-12-23T16:58:04.965+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-53 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:04.965+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:05.125+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:05.125+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:05-54999f1d9a8f659f34bfb476", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353885125), what: "moveChunk.from", ns: "benchdb1.COL-53", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 455, step 3 of 6: 1, step 4 of 6: 20, step 5 of 6: 268, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:05.181+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1800df49298418ff40') } ntoreturn:1 keyUpdates:0 reslen:37 962ms 2014-12-23T16:58:05.182+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:05.183+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1800df49298418ff40') } 2014-12-23T16:58:05.482+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f1d9a8f659f34bfb477 2014-12-23T16:58:05.482+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:05-54999f1d9a8f659f34bfb478", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353885482), what: "moveChunk.start", ns: "benchdb1.COL-53", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:05.534+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-53 based on current shard version 2|0||54999f1800df49298418ff40, current metadata version is 2|0||54999f1800df49298418ff40 2014-12-23T16:58:05.535+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-53 from shard version 2|0||54999f1800df49298418ff40 to shard version 2|1||54999f1800df49298418ff40 2014-12-23T16:58:05.535+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f1800df49298418ff40, took 0ms 2014-12-23T16:58:05.535+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f1800df49298418ff40 2014-12-23T16:58:05.535+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:05.537+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:05.540+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:05.544+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:05.553+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:05.569+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:05.569+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:05.570+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:05.570+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f1800df49298418ff40 2014-12-23T16:58:05.622+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:05.622+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f1800df49298418ff40 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-53' 2014-12-23T16:58:05.878+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:05-54999f1d9a8f659f34bfb479", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353885878), what: "moveChunk.commit", ns: "benchdb1.COL-53", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:05.959+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:05.959+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:05.959+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-53 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1549 2014-12-23T16:58:05.959+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-53 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:05.959+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:06.213+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:06.213+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:06-54999f1e9a8f659f34bfb47a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353886213), what: "moveChunk.from", ns: "benchdb1.COL-53", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 352, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 389, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:06.278+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1800df49298418ff40') } ntoreturn:1 keyUpdates:0 reslen:37 1095ms 2014-12-23T16:58:06.281+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-53", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-53-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f1800df49298418ff40') } 2014-12-23T16:58:06.328+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:58:05 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:58:06.888+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f1e9a8f659f34bfb47b 2014-12-23T16:58:06.888+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-53 based on current shard version 3|0||54999f1800df49298418ff40, current metadata version is 3|0||54999f1800df49298418ff40 2014-12-23T16:58:06.889+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-53 from shard version 3|0||54999f1800df49298418ff40 to shard version 3|1||54999f1800df49298418ff40 2014-12-23T16:58:06.889+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f1800df49298418ff40, took 0ms 2014-12-23T16:58:06.889+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f1800df49298418ff40 2014-12-23T16:58:07.194+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:07-54999f1f9a8f659f34bfb47c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353887194), what: "split", ns: "benchdb1.COL-53", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f1800df49298418ff40') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f1800df49298418ff40') } } } 2014-12-23T16:58:07.564+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:07.564+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-53", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-53-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f1800df49298418ff40') } ntoreturn:1 keyUpdates:0 reslen:37 1283ms 2014-12-23T16:58:10.126+0000 I INDEX [conn2] build index on: benchdb1.COL-54 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-54" } 2014-12-23T16:58:10.126+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:10.127+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:10.478+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-54 with requested shard version 1|2||54999f1f00df49298418ff44, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:10.479+0000 I SHARDING [conn2] collection benchdb1.COL-54 was previously unsharded, new metadata loaded with shard version 1|2||54999f1f00df49298418ff44 2014-12-23T16:58:10.479+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f1f00df49298418ff44, took 0ms 2014-12-23T16:58:10.531+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:10.531+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1f00df49298418ff44') } 2014-12-23T16:58:11.442+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f229a8f659f34bfb47e 2014-12-23T16:58:11.442+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:11-54999f239a8f659f34bfb47f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353891442), what: "moveChunk.start", ns: "benchdb1.COL-54", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:11.510+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-54 based on current shard version 1|2||54999f1f00df49298418ff44, current metadata version is 1|2||54999f1f00df49298418ff44 2014-12-23T16:58:11.511+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-54 already up to date (shard version : 1|2||54999f1f00df49298418ff44, took 0ms) 2014-12-23T16:58:11.511+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f1f00df49298418ff44 2014-12-23T16:58:11.512+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:11.514+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:11.517+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:11.521+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:11.521+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:11.522+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:11.522+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f1f00df49298418ff44 2014-12-23T16:58:11.576+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:11.576+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f1f00df49298418ff44 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-54' 2014-12-23T16:58:11.882+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:11-54999f239a8f659f34bfb480", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353891882), what: "moveChunk.commit", ns: "benchdb1.COL-54", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:11.946+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:11.946+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:11.946+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-54 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1567 2014-12-23T16:58:11.946+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-54 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:11.946+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:12.250+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:12.250+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:12-54999f249a8f659f34bfb481", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353892250), what: "moveChunk.from", ns: "benchdb1.COL-54", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 979, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 424, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:12.302+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1f00df49298418ff44') } ntoreturn:1 keyUpdates:0 reslen:37 1771ms 2014-12-23T16:58:12.304+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:12.304+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1f00df49298418ff44') } 2014-12-23T16:58:12.812+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f249a8f659f34bfb482 2014-12-23T16:58:12.812+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:12-54999f249a8f659f34bfb483", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353892812), what: "moveChunk.start", ns: "benchdb1.COL-54", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:12.887+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-54 based on current shard version 2|0||54999f1f00df49298418ff44, current metadata version is 2|0||54999f1f00df49298418ff44 2014-12-23T16:58:12.887+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-54 from shard version 2|0||54999f1f00df49298418ff44 to shard version 2|1||54999f1f00df49298418ff44 2014-12-23T16:58:12.887+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f1f00df49298418ff44, took 0ms 2014-12-23T16:58:12.887+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f1f00df49298418ff44 2014-12-23T16:58:12.888+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:12.890+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:12.892+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:12.897+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:12.905+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:12.922+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:12.922+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:12.922+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:12.922+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f1f00df49298418ff44 2014-12-23T16:58:12.972+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:12.972+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f1f00df49298418ff44 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-54' 2014-12-23T16:58:13.276+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:13-54999f259a8f659f34bfb484", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353893276), what: "moveChunk.commit", ns: "benchdb1.COL-54", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:13.334+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:13.334+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:13.334+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-54 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1575 2014-12-23T16:58:13.334+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-54 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:13.334+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:13.637+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:13.637+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:13-54999f259a8f659f34bfb485", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353893637), what: "moveChunk.from", ns: "benchdb1.COL-54", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 583, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 412, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:13.699+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f1f00df49298418ff44') } ntoreturn:1 keyUpdates:0 reslen:37 1394ms 2014-12-23T16:58:13.700+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-54", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-54-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f1f00df49298418ff44') } 2014-12-23T16:58:14.259+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f259a8f659f34bfb486 2014-12-23T16:58:14.259+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-54 based on current shard version 3|0||54999f1f00df49298418ff44, current metadata version is 3|0||54999f1f00df49298418ff44 2014-12-23T16:58:14.260+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-54 from shard version 3|0||54999f1f00df49298418ff44 to shard version 3|1||54999f1f00df49298418ff44 2014-12-23T16:58:14.260+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f1f00df49298418ff44, took 0ms 2014-12-23T16:58:14.260+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f1f00df49298418ff44 2014-12-23T16:58:14.565+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:14-54999f269a8f659f34bfb487", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353894565), what: "split", ns: "benchdb1.COL-54", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f1f00df49298418ff44') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f1f00df49298418ff44') } } } 2014-12-23T16:58:14.957+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:14.957+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-54", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-54-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f1f00df49298418ff44') } ntoreturn:1 keyUpdates:0 reslen:37 1256ms 2014-12-23T16:58:16.970+0000 I INDEX [conn2] build index on: benchdb1.COL-55 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-55" } 2014-12-23T16:58:16.970+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:16.971+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:17.257+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-55 with requested shard version 1|2||54999f2500df49298418ff48, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:17.258+0000 I SHARDING [conn2] collection benchdb1.COL-55 was previously unsharded, new metadata loaded with shard version 1|2||54999f2500df49298418ff48 2014-12-23T16:58:17.258+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f2500df49298418ff48, took 0ms 2014-12-23T16:58:17.302+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:17.302+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2500df49298418ff48') } 2014-12-23T16:58:17.730+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f299a8f659f34bfb489 2014-12-23T16:58:17.730+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:17-54999f299a8f659f34bfb48a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353897730), what: "moveChunk.start", ns: "benchdb1.COL-55", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:17.784+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-55 based on current shard version 1|2||54999f2500df49298418ff48, current metadata version is 1|2||54999f2500df49298418ff48 2014-12-23T16:58:17.785+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-55 already up to date (shard version : 1|2||54999f2500df49298418ff48, took 0ms) 2014-12-23T16:58:17.785+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f2500df49298418ff48 2014-12-23T16:58:17.786+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:17.788+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:17.791+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:17.795+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:17.795+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:17.795+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:17.795+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f2500df49298418ff48 2014-12-23T16:58:17.842+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:17.842+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f2500df49298418ff48 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-55' 2014-12-23T16:58:18.046+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:18-54999f2a9a8f659f34bfb48b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353898046), what: "moveChunk.commit", ns: "benchdb1.COL-55", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:18.103+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:18.103+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:18.103+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-55 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1596 2014-12-23T16:58:18.104+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-55 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:18.104+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:18.271+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:18.271+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:18-54999f2a9a8f659f34bfb48c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353898271), what: "moveChunk.from", ns: "benchdb1.COL-55", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 482, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 308, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:18.321+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2500df49298418ff48') } ntoreturn:1 keyUpdates:0 reslen:37 1018ms 2014-12-23T16:58:18.322+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:18.322+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2500df49298418ff48') } 2014-12-23T16:58:18.665+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f2a9a8f659f34bfb48d 2014-12-23T16:58:18.665+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:18-54999f2a9a8f659f34bfb48e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353898665), what: "moveChunk.start", ns: "benchdb1.COL-55", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:18.720+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-55 based on current shard version 2|0||54999f2500df49298418ff48, current metadata version is 2|0||54999f2500df49298418ff48 2014-12-23T16:58:18.720+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-55 from shard version 2|0||54999f2500df49298418ff48 to shard version 2|1||54999f2500df49298418ff48 2014-12-23T16:58:18.720+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f2500df49298418ff48, took 0ms 2014-12-23T16:58:18.720+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f2500df49298418ff48 2014-12-23T16:58:18.721+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:18.723+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:18.725+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:18.730+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:18.738+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:18.755+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:18.755+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:18.755+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:18.755+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f2500df49298418ff48 2014-12-23T16:58:18.809+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:18.809+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f2500df49298418ff48 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-55' 2014-12-23T16:58:19.051+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:19-54999f2b9a8f659f34bfb48f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353899051), what: "moveChunk.commit", ns: "benchdb1.COL-55", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:19.117+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:19.117+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:19.117+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-55 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1604 2014-12-23T16:58:19.117+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-55 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:19.118+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:19.421+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:19.421+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:19-54999f2b9a8f659f34bfb490", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353899421), what: "moveChunk.from", ns: "benchdb1.COL-55", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 398, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 362, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:19.477+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2500df49298418ff48') } ntoreturn:1 keyUpdates:0 reslen:37 1155ms 2014-12-23T16:58:19.479+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-55", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-55-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f2500df49298418ff48') } 2014-12-23T16:58:20.086+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f2b9a8f659f34bfb491 2014-12-23T16:58:20.086+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-55 based on current shard version 3|0||54999f2500df49298418ff48, current metadata version is 3|0||54999f2500df49298418ff48 2014-12-23T16:58:20.087+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-55 from shard version 3|0||54999f2500df49298418ff48 to shard version 3|1||54999f2500df49298418ff48 2014-12-23T16:58:20.087+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f2500df49298418ff48, took 0ms 2014-12-23T16:58:20.087+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f2500df49298418ff48 2014-12-23T16:58:20.392+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:20-54999f2c9a8f659f34bfb492", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353900392), what: "split", ns: "benchdb1.COL-55", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f2500df49298418ff48') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f2500df49298418ff48') } } } 2014-12-23T16:58:20.775+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:20.775+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-55", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-55-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f2500df49298418ff48') } ntoreturn:1 keyUpdates:0 reslen:37 1296ms 2014-12-23T16:58:23.376+0000 I INDEX [conn2] build index on: benchdb1.COL-56 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-56" } 2014-12-23T16:58:23.376+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:23.377+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:23.691+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-56 with requested shard version 1|2||54999f2c00df49298418ff4c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:23.692+0000 I SHARDING [conn2] collection benchdb1.COL-56 was previously unsharded, new metadata loaded with shard version 1|2||54999f2c00df49298418ff4c 2014-12-23T16:58:23.692+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f2c00df49298418ff4c, took 0ms 2014-12-23T16:58:23.740+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:23.740+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2c00df49298418ff4c') } 2014-12-23T16:58:24.650+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f309a8f659f34bfb494 2014-12-23T16:58:24.650+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:24-54999f309a8f659f34bfb495", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353904650), what: "moveChunk.start", ns: "benchdb1.COL-56", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:24.709+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-56 based on current shard version 1|2||54999f2c00df49298418ff4c, current metadata version is 1|2||54999f2c00df49298418ff4c 2014-12-23T16:58:24.710+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-56 already up to date (shard version : 1|2||54999f2c00df49298418ff4c, took 0ms) 2014-12-23T16:58:24.710+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f2c00df49298418ff4c 2014-12-23T16:58:24.710+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:24.713+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:24.716+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:24.720+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:24.720+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:24.720+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:24.720+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f2c00df49298418ff4c 2014-12-23T16:58:24.773+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:24.774+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f2c00df49298418ff4c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-56' 2014-12-23T16:58:25.078+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:25-54999f319a8f659f34bfb496", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353905078), what: "moveChunk.commit", ns: "benchdb1.COL-56", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:25.128+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:25.128+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:25.128+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-56 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1623 2014-12-23T16:58:25.128+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-56 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:25.128+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:25.431+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:25.431+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:25-54999f319a8f659f34bfb497", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353905431), what: "moveChunk.from", ns: "benchdb1.COL-56", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 969, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 408, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:25.481+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2c00df49298418ff4c') } ntoreturn:1 keyUpdates:0 reslen:37 1741ms 2014-12-23T16:58:25.482+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:25.483+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2c00df49298418ff4c') } 2014-12-23T16:58:26.090+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f319a8f659f34bfb498 2014-12-23T16:58:26.090+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:26-54999f329a8f659f34bfb499", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353906090), what: "moveChunk.start", ns: "benchdb1.COL-56", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:26.142+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-56 based on current shard version 2|0||54999f2c00df49298418ff4c, current metadata version is 2|0||54999f2c00df49298418ff4c 2014-12-23T16:58:26.143+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-56 from shard version 2|0||54999f2c00df49298418ff4c to shard version 2|1||54999f2c00df49298418ff4c 2014-12-23T16:58:26.143+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f2c00df49298418ff4c, took 0ms 2014-12-23T16:58:26.143+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f2c00df49298418ff4c 2014-12-23T16:58:26.143+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:26.145+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:26.148+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:26.152+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:26.160+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:26.177+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:26.177+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:26.177+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:26.177+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f2c00df49298418ff4c 2014-12-23T16:58:26.234+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:26.234+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f2c00df49298418ff4c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-56' 2014-12-23T16:58:26.539+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:26-54999f329a8f659f34bfb49a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353906539), what: "moveChunk.commit", ns: "benchdb1.COL-56", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:26.594+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:26.594+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:26.594+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-56 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1631 2014-12-23T16:58:26.594+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-56 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:26.594+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:26.897+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:26.897+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:26-54999f329a8f659f34bfb49b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353906897), what: "moveChunk.from", ns: "benchdb1.COL-56", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 659, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 416, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:26.957+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f2c00df49298418ff4c') } ntoreturn:1 keyUpdates:0 reslen:37 1474ms 2014-12-23T16:58:26.958+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-56", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-56-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f2c00df49298418ff4c') } 2014-12-23T16:58:27.565+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f329a8f659f34bfb49c 2014-12-23T16:58:27.565+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-56 based on current shard version 3|0||54999f2c00df49298418ff4c, current metadata version is 3|0||54999f2c00df49298418ff4c 2014-12-23T16:58:27.566+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-56 from shard version 3|0||54999f2c00df49298418ff4c to shard version 3|1||54999f2c00df49298418ff4c 2014-12-23T16:58:27.566+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f2c00df49298418ff4c, took 0ms 2014-12-23T16:58:27.566+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f2c00df49298418ff4c 2014-12-23T16:58:27.870+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:27-54999f339a8f659f34bfb49d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353907870), what: "split", ns: "benchdb1.COL-56", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f2c00df49298418ff4c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f2c00df49298418ff4c') } } } 2014-12-23T16:58:28.244+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:28.244+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-56", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-56-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f2c00df49298418ff4c') } ntoreturn:1 keyUpdates:0 reslen:37 1285ms 2014-12-23T16:58:30.282+0000 I INDEX [conn2] build index on: benchdb1.COL-57 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-57" } 2014-12-23T16:58:30.282+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:30.283+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:30.557+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-57 with requested shard version 1|2||54999f3300df49298418ff50, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:30.558+0000 I SHARDING [conn2] collection benchdb1.COL-57 was previously unsharded, new metadata loaded with shard version 1|2||54999f3300df49298418ff50 2014-12-23T16:58:30.558+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f3300df49298418ff50, took 0ms 2014-12-23T16:58:30.604+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:30.604+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3300df49298418ff50') } 2014-12-23T16:58:31.009+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f369a8f659f34bfb49f 2014-12-23T16:58:31.009+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:31-54999f379a8f659f34bfb4a0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353911009), what: "moveChunk.start", ns: "benchdb1.COL-57", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:31.061+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-57 based on current shard version 1|2||54999f3300df49298418ff50, current metadata version is 1|2||54999f3300df49298418ff50 2014-12-23T16:58:31.062+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-57 already up to date (shard version : 1|2||54999f3300df49298418ff50, took 0ms) 2014-12-23T16:58:31.062+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f3300df49298418ff50 2014-12-23T16:58:31.063+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:31.065+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.068+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.072+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.072+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:31.072+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:31.072+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f3300df49298418ff50 2014-12-23T16:58:31.120+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:31.120+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f3300df49298418ff50 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-57' 2014-12-23T16:58:31.304+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:31-54999f379a8f659f34bfb4a1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353911304), what: "moveChunk.commit", ns: "benchdb1.COL-57", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:31.357+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:31.357+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:31.357+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-57 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1652 2014-12-23T16:58:31.357+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-57 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:31.357+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:31.519+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:31.520+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:31-54999f379a8f659f34bfb4a2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353911519), what: "moveChunk.from", ns: "benchdb1.COL-57", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 458, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 285, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:31.571+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3300df49298418ff50') } ntoreturn:1 keyUpdates:0 reslen:37 967ms 2014-12-23T16:58:31.573+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:31.573+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3300df49298418ff50') } 2014-12-23T16:58:31.845+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f379a8f659f34bfb4a3 2014-12-23T16:58:31.845+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:31-54999f379a8f659f34bfb4a4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353911845), what: "moveChunk.start", ns: "benchdb1.COL-57", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:31.895+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-57 based on current shard version 2|0||54999f3300df49298418ff50, current metadata version is 2|0||54999f3300df49298418ff50 2014-12-23T16:58:31.896+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-57 from shard version 2|0||54999f3300df49298418ff50 to shard version 2|1||54999f3300df49298418ff50 2014-12-23T16:58:31.896+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f3300df49298418ff50, took 0ms 2014-12-23T16:58:31.896+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f3300df49298418ff50 2014-12-23T16:58:31.896+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:31.898+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.901+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.905+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.914+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.930+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:31.930+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:31.931+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:31.931+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f3300df49298418ff50 2014-12-23T16:58:31.975+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:31.975+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f3300df49298418ff50 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-57' 2014-12-23T16:58:32.113+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:32-54999f389a8f659f34bfb4a5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353912113), what: "moveChunk.commit", ns: "benchdb1.COL-57", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:32.177+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:32.177+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:32.177+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-57 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1660 2014-12-23T16:58:32.177+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-57 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:32.177+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:32.383+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:32.383+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:32-54999f389a8f659f34bfb4a6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353912383), what: "moveChunk.from", ns: "benchdb1.COL-57", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 322, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 246, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:32.458+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3300df49298418ff50') } ntoreturn:1 keyUpdates:0 reslen:37 885ms 2014-12-23T16:58:32.460+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-57", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-57-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f3300df49298418ff50') } 2014-12-23T16:58:33.067+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f389a8f659f34bfb4a7 2014-12-23T16:58:33.067+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-57 based on current shard version 3|0||54999f3300df49298418ff50, current metadata version is 3|0||54999f3300df49298418ff50 2014-12-23T16:58:33.068+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-57 from shard version 3|0||54999f3300df49298418ff50 to shard version 3|1||54999f3300df49298418ff50 2014-12-23T16:58:33.068+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f3300df49298418ff50, took 0ms 2014-12-23T16:58:33.068+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f3300df49298418ff50 2014-12-23T16:58:33.372+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:33-54999f399a8f659f34bfb4a8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353913372), what: "split", ns: "benchdb1.COL-57", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f3300df49298418ff50') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f3300df49298418ff50') } } } 2014-12-23T16:58:33.745+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:33.746+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-57", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-57-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f3300df49298418ff50') } ntoreturn:1 keyUpdates:0 reslen:37 1285ms 2014-12-23T16:58:36.340+0000 I INDEX [conn2] build index on: benchdb1.COL-58 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-58" } 2014-12-23T16:58:36.340+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:36.341+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:36.643+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-58 with requested shard version 1|2||54999f3900df49298418ff54, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:36.644+0000 I SHARDING [conn2] collection benchdb1.COL-58 was previously unsharded, new metadata loaded with shard version 1|2||54999f3900df49298418ff54 2014-12-23T16:58:36.644+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f3900df49298418ff54, took 0ms 2014-12-23T16:58:36.690+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:36.691+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3900df49298418ff54') } 2014-12-23T16:58:36.740+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:58:36 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:58:37.600+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f3c9a8f659f34bfb4aa 2014-12-23T16:58:37.600+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:37-54999f3d9a8f659f34bfb4ab", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353917600), what: "moveChunk.start", ns: "benchdb1.COL-58", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:37.660+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-58 based on current shard version 1|2||54999f3900df49298418ff54, current metadata version is 1|2||54999f3900df49298418ff54 2014-12-23T16:58:37.661+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-58 already up to date (shard version : 1|2||54999f3900df49298418ff54, took 0ms) 2014-12-23T16:58:37.661+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f3900df49298418ff54 2014-12-23T16:58:37.661+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:37.664+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:37.666+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:37.671+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:37.671+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:37.671+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:37.671+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f3900df49298418ff54 2014-12-23T16:58:37.726+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:37.726+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f3900df49298418ff54 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-58' 2014-12-23T16:58:38.031+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:38-54999f3e9a8f659f34bfb4ac", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353918031), what: "moveChunk.commit", ns: "benchdb1.COL-58", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:38.085+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:38.085+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:38.085+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-58 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1679 2014-12-23T16:58:38.085+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-58 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:38.085+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:38.388+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:38.388+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:38-54999f3e9a8f659f34bfb4ad", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353918388), what: "moveChunk.from", ns: "benchdb1.COL-58", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 969, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:38.441+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3900df49298418ff54') } ntoreturn:1 keyUpdates:0 reslen:37 1750ms 2014-12-23T16:58:38.442+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:38.443+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3900df49298418ff54') } 2014-12-23T16:58:39.001+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f3e9a8f659f34bfb4ae 2014-12-23T16:58:39.001+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:38-54999f3e9a8f659f34bfb4af", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353919001), what: "moveChunk.start", ns: "benchdb1.COL-58", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:39.067+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-58 based on current shard version 2|0||54999f3900df49298418ff54, current metadata version is 2|0||54999f3900df49298418ff54 2014-12-23T16:58:39.068+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-58 from shard version 2|0||54999f3900df49298418ff54 to shard version 2|1||54999f3900df49298418ff54 2014-12-23T16:58:39.068+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f3900df49298418ff54, took 0ms 2014-12-23T16:58:39.068+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f3900df49298418ff54 2014-12-23T16:58:39.068+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:39.070+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:39.073+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:39.077+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:39.086+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:39.102+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:39.102+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:39.103+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:39.103+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f3900df49298418ff54 2014-12-23T16:58:39.152+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:39.152+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f3900df49298418ff54 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-58' 2014-12-23T16:58:39.457+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:39-54999f3f9a8f659f34bfb4b0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353919457), what: "moveChunk.commit", ns: "benchdb1.COL-58", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:39.518+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:39.518+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:39.518+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-58 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1687 2014-12-23T16:58:39.518+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-58 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:39.518+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:39.821+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:39.821+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:39-54999f3f9a8f659f34bfb4b1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353919821), what: "moveChunk.from", ns: "benchdb1.COL-58", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 625, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 415, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:39.875+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f3900df49298418ff54') } ntoreturn:1 keyUpdates:0 reslen:37 1432ms 2014-12-23T16:58:39.877+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-58", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-58-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f3900df49298418ff54') } 2014-12-23T16:58:40.484+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f3f9a8f659f34bfb4b2 2014-12-23T16:58:40.484+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-58 based on current shard version 3|0||54999f3900df49298418ff54, current metadata version is 3|0||54999f3900df49298418ff54 2014-12-23T16:58:40.485+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-58 from shard version 3|0||54999f3900df49298418ff54 to shard version 3|1||54999f3900df49298418ff54 2014-12-23T16:58:40.485+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f3900df49298418ff54, took 0ms 2014-12-23T16:58:40.485+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f3900df49298418ff54 2014-12-23T16:58:40.790+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:40-54999f409a8f659f34bfb4b3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353920790), what: "split", ns: "benchdb1.COL-58", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f3900df49298418ff54') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f3900df49298418ff54') } } } 2014-12-23T16:58:41.164+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:41.164+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-58", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-58-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f3900df49298418ff54') } ntoreturn:1 keyUpdates:0 reslen:37 1287ms 2014-12-23T16:58:43.346+0000 I INDEX [conn2] build index on: benchdb1.COL-59 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-59" } 2014-12-23T16:58:43.346+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:43.347+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:43.662+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-59 with requested shard version 1|2||54999f4000df49298418ff58, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:43.663+0000 I SHARDING [conn2] collection benchdb1.COL-59 was previously unsharded, new metadata loaded with shard version 1|2||54999f4000df49298418ff58 2014-12-23T16:58:43.663+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f4000df49298418ff58, took 0ms 2014-12-23T16:58:43.748+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:43.748+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4000df49298418ff58') } 2014-12-23T16:58:44.145+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f439a8f659f34bfb4b5 2014-12-23T16:58:44.145+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:44-54999f449a8f659f34bfb4b6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353924145), what: "moveChunk.start", ns: "benchdb1.COL-59", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:44.199+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-59 based on current shard version 1|2||54999f4000df49298418ff58, current metadata version is 1|2||54999f4000df49298418ff58 2014-12-23T16:58:44.200+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-59 already up to date (shard version : 1|2||54999f4000df49298418ff58, took 0ms) 2014-12-23T16:58:44.200+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f4000df49298418ff58 2014-12-23T16:58:44.200+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:44.203+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:44.205+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:44.210+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:44.210+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:44.210+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:44.210+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f4000df49298418ff58 2014-12-23T16:58:44.257+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:44.258+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f4000df49298418ff58 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-59' 2014-12-23T16:58:44.455+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:44-54999f449a8f659f34bfb4b7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353924455), what: "moveChunk.commit", ns: "benchdb1.COL-59", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:44.510+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:44.510+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:44.510+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-59 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1708 2014-12-23T16:58:44.510+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-59 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:44.510+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:44.674+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:44.674+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:44-54999f449a8f659f34bfb4b8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353924674), what: "moveChunk.from", ns: "benchdb1.COL-59", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 451, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 300, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:44.727+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4000df49298418ff58') } ntoreturn:1 keyUpdates:0 reslen:37 978ms 2014-12-23T16:58:44.728+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:44.728+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4000df49298418ff58') } 2014-12-23T16:58:45.025+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f449a8f659f34bfb4b9 2014-12-23T16:58:45.025+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:45-54999f459a8f659f34bfb4ba", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353925025), what: "moveChunk.start", ns: "benchdb1.COL-59", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:45.077+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-59 based on current shard version 2|0||54999f4000df49298418ff58, current metadata version is 2|0||54999f4000df49298418ff58 2014-12-23T16:58:45.077+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-59 from shard version 2|0||54999f4000df49298418ff58 to shard version 2|1||54999f4000df49298418ff58 2014-12-23T16:58:45.077+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f4000df49298418ff58, took 0ms 2014-12-23T16:58:45.077+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f4000df49298418ff58 2014-12-23T16:58:45.078+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:45.080+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:45.082+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:45.087+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:45.095+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:45.112+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:45.112+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:45.112+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:45.112+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f4000df49298418ff58 2014-12-23T16:58:45.165+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:45.165+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f4000df49298418ff58 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-59' 2014-12-23T16:58:45.370+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:45-54999f459a8f659f34bfb4bb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353925370), what: "moveChunk.commit", ns: "benchdb1.COL-59", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:45.421+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:45.421+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:45.421+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-59 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1716 2014-12-23T16:58:45.421+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-59 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:45.421+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:45.599+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:45.599+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:45-54999f459a8f659f34bfb4bc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353925599), what: "moveChunk.from", ns: "benchdb1.COL-59", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 349, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 309, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:45.641+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4000df49298418ff58') } ntoreturn:1 keyUpdates:0 reslen:37 913ms 2014-12-23T16:58:45.643+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-59", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-59-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f4000df49298418ff58') } 2014-12-23T16:58:46.201+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f459a8f659f34bfb4bd 2014-12-23T16:58:46.201+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-59 based on current shard version 3|0||54999f4000df49298418ff58, current metadata version is 3|0||54999f4000df49298418ff58 2014-12-23T16:58:46.202+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-59 from shard version 3|0||54999f4000df49298418ff58 to shard version 3|1||54999f4000df49298418ff58 2014-12-23T16:58:46.202+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f4000df49298418ff58, took 0ms 2014-12-23T16:58:46.202+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f4000df49298418ff58 2014-12-23T16:58:46.506+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:46-54999f469a8f659f34bfb4be", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353926506), what: "split", ns: "benchdb1.COL-59", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f4000df49298418ff58') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f4000df49298418ff58') } } } 2014-12-23T16:58:46.881+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:46.881+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-59", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-59-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f4000df49298418ff58') } ntoreturn:1 keyUpdates:0 reslen:37 1238ms 2014-12-23T16:58:49.461+0000 I INDEX [conn2] build index on: benchdb1.COL-60 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-60" } 2014-12-23T16:58:49.461+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:49.462+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:49.791+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-60 with requested shard version 1|2||54999f4600df49298418ff5c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:49.791+0000 I SHARDING [conn2] collection benchdb1.COL-60 was previously unsharded, new metadata loaded with shard version 1|2||54999f4600df49298418ff5c 2014-12-23T16:58:49.791+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f4600df49298418ff5c, took 0ms 2014-12-23T16:58:49.838+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:49.839+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4600df49298418ff5c') } 2014-12-23T16:58:50.702+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f4a9a8f659f34bfb4c0 2014-12-23T16:58:50.702+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:50-54999f4a9a8f659f34bfb4c1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353930702), what: "moveChunk.start", ns: "benchdb1.COL-60", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:50.760+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-60 based on current shard version 1|2||54999f4600df49298418ff5c, current metadata version is 1|2||54999f4600df49298418ff5c 2014-12-23T16:58:50.761+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-60 already up to date (shard version : 1|2||54999f4600df49298418ff5c, took 0ms) 2014-12-23T16:58:50.761+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f4600df49298418ff5c 2014-12-23T16:58:50.762+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:50.764+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:50.767+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:50.771+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:50.771+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:50.771+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:50.771+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f4600df49298418ff5c 2014-12-23T16:58:50.826+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:50.826+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f4600df49298418ff5c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-60' 2014-12-23T16:58:51.131+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:51-54999f4b9a8f659f34bfb4c2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353931131), what: "moveChunk.commit", ns: "benchdb1.COL-60", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:51.189+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:51.189+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:51.189+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-60 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1734 2014-12-23T16:58:51.189+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-60 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:51.189+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:51.492+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:51.492+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:51-54999f4b9a8f659f34bfb4c3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353931492), what: "moveChunk.from", ns: "benchdb1.COL-60", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 922, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 417, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:51.551+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4600df49298418ff5c') } ntoreturn:1 keyUpdates:0 reslen:37 1712ms 2014-12-23T16:58:51.552+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:51.553+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4600df49298418ff5c') } 2014-12-23T16:58:52.160+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f4b9a8f659f34bfb4c4 2014-12-23T16:58:52.160+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:52-54999f4c9a8f659f34bfb4c5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353932160), what: "moveChunk.start", ns: "benchdb1.COL-60", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:52.240+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-60 based on current shard version 2|0||54999f4600df49298418ff5c, current metadata version is 2|0||54999f4600df49298418ff5c 2014-12-23T16:58:52.241+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-60 from shard version 2|0||54999f4600df49298418ff5c to shard version 2|1||54999f4600df49298418ff5c 2014-12-23T16:58:52.241+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f4600df49298418ff5c, took 0ms 2014-12-23T16:58:52.241+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f4600df49298418ff5c 2014-12-23T16:58:52.241+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:52.244+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:52.246+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:52.250+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:52.259+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:52.275+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:52.275+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:52.276+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:52.276+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f4600df49298418ff5c 2014-12-23T16:58:52.330+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:52.330+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f4600df49298418ff5c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-60' 2014-12-23T16:58:52.635+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:52-54999f4c9a8f659f34bfb4c6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353932635), what: "moveChunk.commit", ns: "benchdb1.COL-60", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:52.695+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:52.696+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:52.696+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-60 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1742 2014-12-23T16:58:52.696+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-60 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:52.696+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:52.999+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:52.999+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:52-54999f4c9a8f659f34bfb4c7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353932999), what: "moveChunk.from", ns: "benchdb1.COL-60", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 688, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 420, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:53.056+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4600df49298418ff5c') } ntoreturn:1 keyUpdates:0 reslen:37 1503ms 2014-12-23T16:58:53.057+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-60", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-60-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f4600df49298418ff5c') } 2014-12-23T16:58:53.664+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f4d9a8f659f34bfb4c8 2014-12-23T16:58:53.664+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-60 based on current shard version 3|0||54999f4600df49298418ff5c, current metadata version is 3|0||54999f4600df49298418ff5c 2014-12-23T16:58:53.665+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-60 from shard version 3|0||54999f4600df49298418ff5c to shard version 3|1||54999f4600df49298418ff5c 2014-12-23T16:58:53.665+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f4600df49298418ff5c, took 0ms 2014-12-23T16:58:53.665+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f4600df49298418ff5c 2014-12-23T16:58:53.969+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:53-54999f4d9a8f659f34bfb4c9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353933969), what: "split", ns: "benchdb1.COL-60", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f4600df49298418ff5c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f4600df49298418ff5c') } } } 2014-12-23T16:58:54.350+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:54.350+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-60", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-60-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f4600df49298418ff5c') } ntoreturn:1 keyUpdates:0 reslen:37 1292ms 2014-12-23T16:58:56.618+0000 I INDEX [conn2] build index on: benchdb1.COL-61 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-61" } 2014-12-23T16:58:56.618+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:58:56.619+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:58:56.898+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-61 with requested shard version 1|2||54999f4d00df49298418ff60, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:58:56.899+0000 I SHARDING [conn2] collection benchdb1.COL-61 was previously unsharded, new metadata loaded with shard version 1|2||54999f4d00df49298418ff60 2014-12-23T16:58:56.899+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f4d00df49298418ff60, took 0ms 2014-12-23T16:58:56.940+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:56.941+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4d00df49298418ff60') } 2014-12-23T16:58:57.377+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f519a8f659f34bfb4cb 2014-12-23T16:58:57.377+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:57-54999f519a8f659f34bfb4cc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353937377), what: "moveChunk.start", ns: "benchdb1.COL-61", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:58:57.427+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-61 based on current shard version 1|2||54999f4d00df49298418ff60, current metadata version is 1|2||54999f4d00df49298418ff60 2014-12-23T16:58:57.428+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-61 already up to date (shard version : 1|2||54999f4d00df49298418ff60, took 0ms) 2014-12-23T16:58:57.428+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||54999f4d00df49298418ff60 2014-12-23T16:58:57.429+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:57.431+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:57.434+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:57.438+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:57.438+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:57.438+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:57.439+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||54999f4d00df49298418ff60 2014-12-23T16:58:57.487+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:57.487+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||54999f4d00df49298418ff60 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-61' 2014-12-23T16:58:57.696+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:57-54999f519a8f659f34bfb4cd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353937696), what: "moveChunk.commit", ns: "benchdb1.COL-61", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:57.747+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:57.748+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:57.748+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-61 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1763 2014-12-23T16:58:57.748+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-61 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:58:57.748+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:57.902+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:57.903+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:57-54999f519a8f659f34bfb4ce", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353937903), what: "moveChunk.from", ns: "benchdb1.COL-61", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 487, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 309, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:58:57.955+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4d00df49298418ff60') } ntoreturn:1 keyUpdates:0 reslen:37 1014ms 2014-12-23T16:58:57.956+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T16:58:57.956+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4d00df49298418ff60') } 2014-12-23T16:58:58.245+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f519a8f659f34bfb4cf 2014-12-23T16:58:58.246+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:58-54999f529a8f659f34bfb4d0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353938246), what: "moveChunk.start", ns: "benchdb1.COL-61", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:58:58.296+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-61 based on current shard version 2|0||54999f4d00df49298418ff60, current metadata version is 2|0||54999f4d00df49298418ff60 2014-12-23T16:58:58.297+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-61 from shard version 2|0||54999f4d00df49298418ff60 to shard version 2|1||54999f4d00df49298418ff60 2014-12-23T16:58:58.297+0000 I SHARDING [conn4] collection version was loaded at version 2|1||54999f4d00df49298418ff60, took 0ms 2014-12-23T16:58:58.297+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||54999f4d00df49298418ff60 2014-12-23T16:58:58.297+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T16:58:58.300+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:58.302+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:58.307+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:58.315+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:58.331+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:58:58.332+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T16:58:58.332+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T16:58:58.332+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||54999f4d00df49298418ff60 2014-12-23T16:58:58.395+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:58:58.396+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||54999f4d00df49298418ff60 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-61' 2014-12-23T16:58:58.591+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:58-54999f529a8f659f34bfb4d1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353938591), what: "moveChunk.commit", ns: "benchdb1.COL-61", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:58:58.646+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:58.646+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T16:58:58.646+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-61 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1771 2014-12-23T16:58:58.646+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-61 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:58:58.646+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:58:58.811+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:58.811+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:58-54999f529a8f659f34bfb4d2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353938811), what: "moveChunk.from", ns: "benchdb1.COL-61", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 340, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 314, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:58:58.868+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f4d00df49298418ff60') } ntoreturn:1 keyUpdates:0 reslen:37 911ms 2014-12-23T16:58:58.870+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-61", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-61-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f4d00df49298418ff60') } 2014-12-23T16:58:59.309+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f529a8f659f34bfb4d3 2014-12-23T16:58:59.309+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-61 based on current shard version 3|0||54999f4d00df49298418ff60, current metadata version is 3|0||54999f4d00df49298418ff60 2014-12-23T16:58:59.310+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-61 from shard version 3|0||54999f4d00df49298418ff60 to shard version 3|1||54999f4d00df49298418ff60 2014-12-23T16:58:59.310+0000 I SHARDING [conn4] collection version was loaded at version 3|1||54999f4d00df49298418ff60, took 0ms 2014-12-23T16:58:59.310+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||54999f4d00df49298418ff60 2014-12-23T16:58:59.526+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:58:59-54999f539a8f659f34bfb4d4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419353939526), what: "split", ns: "benchdb1.COL-61", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f4d00df49298418ff60') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f4d00df49298418ff60') } } } 2014-12-23T16:58:59.913+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:58:59.913+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-61", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-61-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f4d00df49298418ff60') } ntoreturn:1 keyUpdates:0 reslen:37 1043ms 2014-12-23T16:59:02.505+0000 I INDEX [conn2] build index on: benchdb1.COL-62 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-62" } 2014-12-23T16:59:02.505+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:59:02.506+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:59:02.813+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-62 with requested shard version 1|2||54999f5300df49298418ff64, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:59:02.814+0000 I SHARDING [conn2] collection benchdb1.COL-62 was previously unsharded, new metadata loaded with shard version 1|2||54999f5300df49298418ff64 2014-12-23T16:59:02.814+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f5300df49298418ff64, took 1ms 2014-12-23T16:59:02.866+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:59:02.867+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5300df49298418ff64') } 2014-12-23T16:59:03.778+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f579a8f659f34bfb4d6 2014-12-23T16:59:03.778+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:03-54999f579a8f659f34bfb4d7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353943778), what: "moveChunk.start", ns: "benchdb1.COL-62", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:59:03.865+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 based on current shard version 1|2||54999f5300df49298418ff64, current metadata version is 1|2||54999f5300df49298418ff64 2014-12-23T16:59:03.866+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-62 already up to date (shard version : 1|2||54999f5300df49298418ff64, took 0ms) 2014-12-23T16:59:03.866+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999f5300df49298418ff64 2014-12-23T16:59:03.866+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:59:03.869+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:03.872+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:03.876+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:03.876+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:59:03.876+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:59:03.877+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999f5300df49298418ff64 2014-12-23T16:59:03.934+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:59:03.934+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999f5300df49298418ff64 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-62' 2014-12-23T16:59:04.239+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:04-54999f589a8f659f34bfb4d8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353944239), what: "moveChunk.commit", ns: "benchdb1.COL-62", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:59:04.303+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:04.303+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:59:04.303+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-62 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1789 2014-12-23T16:59:04.303+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-62 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:59:04.303+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:04.606+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:59:04.607+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:04-54999f589a8f659f34bfb4d9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353944607), what: "moveChunk.from", ns: "benchdb1.COL-62", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 999, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 426, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:59:04.667+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5300df49298418ff64') } ntoreturn:1 keyUpdates:0 reslen:37 1801ms 2014-12-23T16:59:04.669+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:59:04.669+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5300df49298418ff64') } 2014-12-23T16:59:05.277+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f589a8f659f34bfb4da 2014-12-23T16:59:05.277+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:05-54999f599a8f659f34bfb4db", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353945277), what: "moveChunk.start", ns: "benchdb1.COL-62", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:59:05.338+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 based on current shard version 2|0||54999f5300df49298418ff64, current metadata version is 2|0||54999f5300df49298418ff64 2014-12-23T16:59:05.339+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-62 from shard version 2|0||54999f5300df49298418ff64 to shard version 2|1||54999f5300df49298418ff64 2014-12-23T16:59:05.339+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999f5300df49298418ff64, took 0ms 2014-12-23T16:59:05.339+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999f5300df49298418ff64 2014-12-23T16:59:05.339+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:59:05.341+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:05.344+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:05.348+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:05.357+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:05.373+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:05.373+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:59:05.373+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:59:05.373+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999f5300df49298418ff64 2014-12-23T16:59:05.429+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:59:05.429+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999f5300df49298418ff64 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-62' 2014-12-23T16:59:05.734+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:05-54999f599a8f659f34bfb4dc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353945734), what: "moveChunk.commit", ns: "benchdb1.COL-62", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:59:05.799+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:05.799+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:59:05.800+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-62 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1797 2014-12-23T16:59:05.800+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-62 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:59:05.800+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:06.103+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:59:06.103+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:06-54999f5a9a8f659f34bfb4dd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353946103), what: "moveChunk.from", ns: "benchdb1.COL-62", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 426, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:59:06.159+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5300df49298418ff64') } ntoreturn:1 keyUpdates:0 reslen:37 1490ms 2014-12-23T16:59:06.161+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-62", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-62-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f5300df49298418ff64') } 2014-12-23T16:59:06.769+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f5a9a8f659f34bfb4de 2014-12-23T16:59:06.769+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 based on current shard version 3|0||54999f5300df49298418ff64, current metadata version is 3|0||54999f5300df49298418ff64 2014-12-23T16:59:06.770+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-62 from shard version 3|0||54999f5300df49298418ff64 to shard version 3|1||54999f5300df49298418ff64 2014-12-23T16:59:06.770+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999f5300df49298418ff64, took 0ms 2014-12-23T16:59:06.770+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999f5300df49298418ff64 2014-12-23T16:59:07.076+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:07-54999f5b9a8f659f34bfb4df", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353947076), what: "split", ns: "benchdb1.COL-62", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f5300df49298418ff64') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f5300df49298418ff64') } } } 2014-12-23T16:59:07.206+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:59:06 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T16:59:07.459+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:59:07.460+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-62", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-62-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f5300df49298418ff64') } ntoreturn:1 keyUpdates:0 reslen:37 1298ms 2014-12-23T16:59:09.827+0000 I INDEX [conn2] build index on: benchdb1.COL-63 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-63" } 2014-12-23T16:59:09.827+0000 I INDEX [conn2] building index using bulk method 2014-12-23T16:59:09.828+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T16:59:10.147+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-63 with requested shard version 1|2||54999f5a00df49298418ff68, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T16:59:10.147+0000 I SHARDING [conn2] collection benchdb1.COL-63 was previously unsharded, new metadata loaded with shard version 1|2||54999f5a00df49298418ff68 2014-12-23T16:59:10.147+0000 I SHARDING [conn2] collection version was loaded at version 1|2||54999f5a00df49298418ff68, took 0ms 2014-12-23T16:59:10.190+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:59:10.191+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5a00df49298418ff68') } 2014-12-23T16:59:10.622+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f5e9a8f659f34bfb4e1 2014-12-23T16:59:10.622+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:10-54999f5e9a8f659f34bfb4e2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353950622), what: "moveChunk.start", ns: "benchdb1.COL-63", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T16:59:10.673+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 based on current shard version 1|2||54999f5a00df49298418ff68, current metadata version is 1|2||54999f5a00df49298418ff68 2014-12-23T16:59:10.673+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-63 already up to date (shard version : 1|2||54999f5a00df49298418ff68, took 0ms) 2014-12-23T16:59:10.673+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||54999f5a00df49298418ff68 2014-12-23T16:59:10.674+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:59:10.677+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:10.679+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:10.683+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:10.683+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:59:10.684+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:59:10.684+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||54999f5a00df49298418ff68 2014-12-23T16:59:10.729+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:59:10.729+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||54999f5a00df49298418ff68 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-63' 2014-12-23T16:59:10.928+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:10-54999f5e9a8f659f34bfb4e3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353950928), what: "moveChunk.commit", ns: "benchdb1.COL-63", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:59:10.983+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:10.983+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:59:10.983+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-63 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 1818 2014-12-23T16:59:10.983+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-63 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T16:59:10.983+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:11.175+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:59:11.175+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:11-54999f5f9a8f659f34bfb4e4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353951175), what: "moveChunk.from", ns: "benchdb1.COL-63", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 482, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 299, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T16:59:11.239+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5a00df49298418ff68') } ntoreturn:1 keyUpdates:0 reslen:37 1049ms 2014-12-23T16:59:11.241+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T16:59:11.241+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5a00df49298418ff68') } 2014-12-23T16:59:11.549+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f5f9a8f659f34bfb4e5 2014-12-23T16:59:11.549+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:11-54999f5f9a8f659f34bfb4e6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353951549), what: "moveChunk.start", ns: "benchdb1.COL-63", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T16:59:11.597+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 based on current shard version 2|0||54999f5a00df49298418ff68, current metadata version is 2|0||54999f5a00df49298418ff68 2014-12-23T16:59:11.597+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-63 from shard version 2|0||54999f5a00df49298418ff68 to shard version 2|1||54999f5a00df49298418ff68 2014-12-23T16:59:11.597+0000 I SHARDING [conn1] collection version was loaded at version 2|1||54999f5a00df49298418ff68, took 0ms 2014-12-23T16:59:11.598+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||54999f5a00df49298418ff68 2014-12-23T16:59:11.598+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T16:59:11.600+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:11.603+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:11.607+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:11.615+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:11.632+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T16:59:11.632+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T16:59:11.632+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T16:59:11.632+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||54999f5a00df49298418ff68 2014-12-23T16:59:11.675+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T16:59:11.675+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||54999f5a00df49298418ff68 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-63' 2014-12-23T16:59:11.865+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:11-54999f5f9a8f659f34bfb4e7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353951865), what: "moveChunk.commit", ns: "benchdb1.COL-63", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T16:59:11.919+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:11.919+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T16:59:11.919+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-63 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 1826 2014-12-23T16:59:11.920+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-63 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T16:59:11.920+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T16:59:12.089+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:59:12.089+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:12-54999f609a8f659f34bfb4e8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353952089), what: "moveChunk.from", ns: "benchdb1.COL-63", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 356, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 287, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T16:59:12.148+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('54999f5a00df49298418ff68') } ntoreturn:1 keyUpdates:0 reslen:37 906ms 2014-12-23T16:59:12.149+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-63", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-63-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f5a00df49298418ff68') } 2014-12-23T16:59:12.463+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 54999f609a8f659f34bfb4e9 2014-12-23T16:59:12.463+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 based on current shard version 3|0||54999f5a00df49298418ff68, current metadata version is 3|0||54999f5a00df49298418ff68 2014-12-23T16:59:12.464+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-63 from shard version 3|0||54999f5a00df49298418ff68 to shard version 3|1||54999f5a00df49298418ff68 2014-12-23T16:59:12.464+0000 I SHARDING [conn1] collection version was loaded at version 3|1||54999f5a00df49298418ff68, took 0ms 2014-12-23T16:59:12.464+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||54999f5a00df49298418ff68 2014-12-23T16:59:12.663+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T16:59:12-54999f609a8f659f34bfb4ea", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419353952663), what: "split", ns: "benchdb1.COL-63", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('54999f5a00df49298418ff68') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('54999f5a00df49298418ff68') } } } 2014-12-23T16:59:12.921+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T16:59:12.921+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-63", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-63-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('54999f5a00df49298418ff68') } ntoreturn:1 keyUpdates:0 reslen:37 771ms 2014-12-23T16:59:37.476+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 16:59:37 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:00:08.082+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:00:07 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:00:38.688+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:00:38 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:01:08.947+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:01:08 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:01:39.554+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:01:38 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:02:10.127+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:02:09 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:02:40.437+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:02:40 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:03:11.009+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:03:10 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:03:41.615+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:03:41 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:04:12.172+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:04:11 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:04:42.779+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:04:42 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:05:13.365+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:05:12 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:05:30.118+0000 I INDEX [conn2] build index on: foodb.testc properties: { v: 1, key: { _id: "hashed" }, name: "_id_hashed", ns: "foodb.testc" } 2014-12-23T17:05:30.118+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:05:30.118+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:05:30.386+0000 I SHARDING [conn2] remotely refreshing metadata for foodb.testc with requested shard version 1|2||5499a0d700df49298418ffa6, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:05:30.387+0000 I SHARDING [conn2] collection foodb.testc was previously unsharded, new metadata loaded with shard version 1|2||5499a0d700df49298418ffa6 2014-12-23T17:05:30.387+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a0d700df49298418ffa6, took 0ms 2014-12-23T17:05:30.432+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:05:30.433+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "foodb.testc-_id_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a0d700df49298418ffa6') } 2014-12-23T17:05:31.314+0000 I SHARDING [conn1] distributed lock 'foodb.testc/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a0da9a8f659f34bfb4ec 2014-12-23T17:05:31.314+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:05:31-5499a0db9a8f659f34bfb4ed", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354331314), what: "moveChunk.start", ns: "foodb.testc", details: { min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:05:31.374+0000 I SHARDING [conn1] remotely refreshing metadata for foodb.testc based on current shard version 1|2||5499a0d700df49298418ffa6, current metadata version is 1|2||5499a0d700df49298418ffa6 2014-12-23T17:05:31.375+0000 I SHARDING [conn1] metadata of collection foodb.testc already up to date (shard version : 1|2||5499a0d700df49298418ffa6, took 0ms) 2014-12-23T17:05:31.375+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a0d700df49298418ffa6 2014-12-23T17:05:31.375+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:05:31.378+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, shardKeyPattern: { _id: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:31.380+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, shardKeyPattern: { _id: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:31.385+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, shardKeyPattern: { _id: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:31.385+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:05:31.385+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:05:31.385+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a0d700df49298418ffa6 2014-12-23T17:05:31.434+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, shardKeyPattern: { _id: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:05:31.434+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a0d700df49298418ffa6 through { _id: MinKey } -> { _id: -3074457345618258602 } for collection 'foodb.testc' 2014-12-23T17:05:31.738+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:05:31-5499a0db9a8f659f34bfb4ee", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354331738), what: "moveChunk.commit", ns: "foodb.testc", details: { min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:05:31.796+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:05:31.796+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:05:31.796+0000 I SHARDING [conn1] Deleter starting delete for: foodb.testc from { _id: -3074457345618258602 } -> { _id: 3074457345618258602 }, with opId: 1908 2014-12-23T17:05:31.796+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for foodb.testc from { _id: -3074457345618258602 } -> { _id: 3074457345618258602 } 2014-12-23T17:05:31.796+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:05:32.101+0000 I SHARDING [conn1] distributed lock 'foodb.testc/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:05:32.101+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:05:32-5499a0dc9a8f659f34bfb4ef", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354332101), what: "moveChunk.from", ns: "foodb.testc", details: { min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 941, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 411, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:05:32.157+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { _id: -3074457345618258602 }, max: { _id: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "foodb.testc-_id_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a0d700df49298418ffa6') } ntoreturn:1 keyUpdates:0 reslen:37 1724ms 2014-12-23T17:05:32.159+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:05:32.159+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "foodb.testc-_id_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a0d700df49298418ffa6') } 2014-12-23T17:05:32.774+0000 I SHARDING [conn1] distributed lock 'foodb.testc/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a0dc9a8f659f34bfb4f0 2014-12-23T17:05:32.774+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:05:32-5499a0dc9a8f659f34bfb4f1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354332774), what: "moveChunk.start", ns: "foodb.testc", details: { min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:05:32.829+0000 I SHARDING [conn1] remotely refreshing metadata for foodb.testc based on current shard version 2|0||5499a0d700df49298418ffa6, current metadata version is 2|0||5499a0d700df49298418ffa6 2014-12-23T17:05:32.830+0000 I SHARDING [conn1] updating metadata for foodb.testc from shard version 2|0||5499a0d700df49298418ffa6 to shard version 2|1||5499a0d700df49298418ffa6 2014-12-23T17:05:32.830+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a0d700df49298418ffa6, took 0ms 2014-12-23T17:05:32.830+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a0d700df49298418ffa6 2014-12-23T17:05:32.830+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:05:32.832+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, shardKeyPattern: { _id: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:32.835+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, shardKeyPattern: { _id: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:32.839+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, shardKeyPattern: { _id: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:32.848+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, shardKeyPattern: { _id: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:32.864+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, shardKeyPattern: { _id: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:05:32.864+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:05:32.864+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:05:32.864+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a0d700df49298418ffa6 2014-12-23T17:05:32.918+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, shardKeyPattern: { _id: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:05:32.918+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a0d700df49298418ffa6 through { _id: MinKey } -> { _id: -3074457345618258602 } for collection 'foodb.testc' 2014-12-23T17:05:33.222+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:05:33-5499a0dd9a8f659f34bfb4f2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354333222), what: "moveChunk.commit", ns: "foodb.testc", details: { min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:05:33.301+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:05:33.301+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:05:33.301+0000 I SHARDING [conn1] Deleter starting delete for: foodb.testc from { _id: 3074457345618258602 } -> { _id: MaxKey }, with opId: 1916 2014-12-23T17:05:33.301+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for foodb.testc from { _id: 3074457345618258602 } -> { _id: MaxKey } 2014-12-23T17:05:33.301+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:05:33.606+0000 I SHARDING [conn1] distributed lock 'foodb.testc/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:05:33.606+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:05:33-5499a0dd9a8f659f34bfb4f3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354333606), what: "moveChunk.from", ns: "foodb.testc", details: { min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 436, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:05:33.668+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "foodb.testc", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { _id: 3074457345618258602 }, max: { _id: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "foodb.testc-_id_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a0d700df49298418ffa6') } ntoreturn:1 keyUpdates:0 reslen:37 1509ms 2014-12-23T17:05:33.670+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "foodb.testc", keyPattern: { _id: "hashed" }, min: { _id: MinKey }, max: { _id: -3074457345618258602 }, from: "shard0002", splitKeys: [ { _id: -6148914691236517204 } ], shardId: "foodb.testc-_id_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a0d700df49298418ffa6') } 2014-12-23T17:05:34.277+0000 I SHARDING [conn1] distributed lock 'foodb.testc/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a0dd9a8f659f34bfb4f4 2014-12-23T17:05:34.277+0000 I SHARDING [conn1] remotely refreshing metadata for foodb.testc based on current shard version 3|0||5499a0d700df49298418ffa6, current metadata version is 3|0||5499a0d700df49298418ffa6 2014-12-23T17:05:34.278+0000 I SHARDING [conn1] updating metadata for foodb.testc from shard version 3|0||5499a0d700df49298418ffa6 to shard version 3|1||5499a0d700df49298418ffa6 2014-12-23T17:05:34.278+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a0d700df49298418ffa6, took 0ms 2014-12-23T17:05:34.278+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a0d700df49298418ffa6 2014-12-23T17:05:34.583+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:05:34-5499a0de9a8f659f34bfb4f5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354334583), what: "split", ns: "foodb.testc", details: { before: { min: { _id: MinKey }, max: { _id: -3074457345618258602 } }, left: { min: { _id: MinKey }, max: { _id: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a0d700df49298418ffa6') }, right: { min: { _id: -6148914691236517204 }, max: { _id: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a0d700df49298418ffa6') } } } 2014-12-23T17:05:34.971+0000 I SHARDING [conn1] distributed lock 'foodb.testc/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:05:34.972+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "foodb.testc", keyPattern: { _id: "hashed" }, min: { _id: MinKey }, max: { _id: -3074457345618258602 }, from: "shard0002", splitKeys: [ { _id: -6148914691236517204 } ], shardId: "foodb.testc-_id_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a0d700df49298418ffa6') } ntoreturn:1 keyUpdates:0 reslen:37 1301ms 2014-12-23T17:05:43.972+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:05:43 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:06:05.356+0000 I COMMAND [conn4] CMD: drop foodb.testc 2014-12-23T17:06:05.436+0000 I SHARDING [conn4] remotely refreshing metadata for foodb.testc with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a0d700df49298418ffa6, current metadata version is 3|3||5499a0d700df49298418ffa6 2014-12-23T17:06:05.437+0000 W SHARDING [conn4] no chunks found when reloading foodb.testc, previous version was 0|0||5499a0d700df49298418ffa6, this is a drop 2014-12-23T17:06:05.437+0000 I SHARDING [conn4] dropping metadata for foodb.testc at shard version 3|3||5499a0d700df49298418ffa6, took 0ms 2014-12-23T17:06:14.580+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:06:13 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:06:44.842+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:06:44 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:07:15.448+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:07:14 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:07:46.055+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:07:45 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:08:16.314+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:08:16 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:08:46.921+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:08:46 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:09:17.528+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:09:16 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:09:48.067+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:09:47 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:10:18.675+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:10:18 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:10:49.282+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:10:48 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:11:19.830+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:11:19 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:11:50.436+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:11:49 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:12:21.015+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:12:20 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:12:51.621+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:12:51 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:13:22.228+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:13:21 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:13:52.711+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:13:52 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:14:23.318+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:14:22 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:14:48.959+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-0 2014-12-23T17:14:49.025+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999dad00df49298418fe62, current metadata version is 3|3||54999dad00df49298418fe62 2014-12-23T17:14:49.025+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-0, previous version was 0|0||54999dad00df49298418fe62, this is a drop 2014-12-23T17:14:49.026+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-0 at shard version 3|3||54999dad00df49298418fe62, took 0ms 2014-12-23T17:14:50.064+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-1 2014-12-23T17:14:50.120+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999db600df49298418fe67, current metadata version is 3|3||54999db600df49298418fe67 2014-12-23T17:14:50.121+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-1, previous version was 0|0||54999db600df49298418fe67, this is a drop 2014-12-23T17:14:50.121+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-1 at shard version 3|3||54999db600df49298418fe67, took 0ms 2014-12-23T17:14:51.165+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-10 2014-12-23T17:14:51.226+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999df200df49298418fe8e, current metadata version is 3|3||54999df200df49298418fe8e 2014-12-23T17:14:51.227+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-10, previous version was 0|0||54999df200df49298418fe8e, this is a drop 2014-12-23T17:14:51.227+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-10 at shard version 3|3||54999df200df49298418fe8e, took 0ms 2014-12-23T17:14:52.285+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-11 2014-12-23T17:14:52.343+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999df900df49298418fe93, current metadata version is 3|3||54999df900df49298418fe93 2014-12-23T17:14:52.343+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-11, previous version was 0|0||54999df900df49298418fe93, this is a drop 2014-12-23T17:14:52.343+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-11 at shard version 3|3||54999df900df49298418fe93, took 0ms 2014-12-23T17:14:53.407+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-12 2014-12-23T17:14:53.491+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e0000df49298418fe97, current metadata version is 3|3||54999e0000df49298418fe97 2014-12-23T17:14:53.492+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-12, previous version was 0|0||54999e0000df49298418fe97, this is a drop 2014-12-23T17:14:53.492+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-12 at shard version 3|3||54999e0000df49298418fe97, took 0ms 2014-12-23T17:14:53.816+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:14:53 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:14:54.576+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-13 2014-12-23T17:14:54.638+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e0700df49298418fe9b, current metadata version is 3|3||54999e0700df49298418fe9b 2014-12-23T17:14:54.639+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-13, previous version was 0|0||54999e0700df49298418fe9b, this is a drop 2014-12-23T17:14:54.639+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-13 at shard version 3|3||54999e0700df49298418fe9b, took 0ms 2014-12-23T17:14:55.592+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-14 2014-12-23T17:14:55.663+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e0e00df49298418fea0, current metadata version is 3|3||54999e0e00df49298418fea0 2014-12-23T17:14:55.664+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-14, previous version was 0|0||54999e0e00df49298418fea0, this is a drop 2014-12-23T17:14:55.664+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-14 at shard version 3|3||54999e0e00df49298418fea0, took 0ms 2014-12-23T17:14:56.709+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-15 2014-12-23T17:14:56.771+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e1600df49298418fea3, current metadata version is 3|3||54999e1600df49298418fea3 2014-12-23T17:14:56.771+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-15, previous version was 0|0||54999e1600df49298418fea3, this is a drop 2014-12-23T17:14:56.771+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-15 at shard version 3|3||54999e1600df49298418fea3, took 0ms 2014-12-23T17:14:57.746+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-16 2014-12-23T17:14:57.816+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e1c00df49298418fea8, current metadata version is 3|3||54999e1c00df49298418fea8 2014-12-23T17:14:57.817+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-16, previous version was 0|0||54999e1c00df49298418fea8, this is a drop 2014-12-23T17:14:57.817+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-16 at shard version 3|3||54999e1c00df49298418fea8, took 0ms 2014-12-23T17:14:58.476+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-17 2014-12-23T17:14:58.504+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e2300df49298418feac, current metadata version is 3|3||54999e2300df49298418feac 2014-12-23T17:14:58.504+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-17, previous version was 0|0||54999e2300df49298418feac, this is a drop 2014-12-23T17:14:58.504+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-17 at shard version 3|3||54999e2300df49298418feac, took 0ms 2014-12-23T17:14:59.123+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-18 2014-12-23T17:14:59.153+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e2b00df49298418feb0, current metadata version is 3|3||54999e2b00df49298418feb0 2014-12-23T17:14:59.153+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-18, previous version was 0|0||54999e2b00df49298418feb0, this is a drop 2014-12-23T17:14:59.154+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-18 at shard version 3|3||54999e2b00df49298418feb0, took 0ms 2014-12-23T17:14:59.761+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-19 2014-12-23T17:14:59.797+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e3100df49298418feb5, current metadata version is 3|3||54999e3100df49298418feb5 2014-12-23T17:14:59.798+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-19, previous version was 0|0||54999e3100df49298418feb5, this is a drop 2014-12-23T17:14:59.798+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-19 at shard version 3|3||54999e3100df49298418feb5, took 0ms 2014-12-23T17:15:00.505+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-2 2014-12-23T17:15:00.566+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-2 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999dbd00df49298418fe6c, current metadata version is 3|3||54999dbd00df49298418fe6c 2014-12-23T17:15:00.566+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-2, previous version was 0|0||54999dbd00df49298418fe6c, this is a drop 2014-12-23T17:15:00.566+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-2 at shard version 3|3||54999dbd00df49298418fe6c, took 0ms 2014-12-23T17:15:01.653+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-20 2014-12-23T17:15:01.720+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e3900df49298418feb8, current metadata version is 3|3||54999e3900df49298418feb8 2014-12-23T17:15:01.721+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-20, previous version was 0|0||54999e3900df49298418feb8, this is a drop 2014-12-23T17:15:01.721+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-20 at shard version 3|3||54999e3900df49298418feb8, took 0ms 2014-12-23T17:15:02.804+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-21 2014-12-23T17:15:02.863+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-21 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e4000df49298418febd, current metadata version is 3|3||54999e4000df49298418febd 2014-12-23T17:15:02.864+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-21, previous version was 0|0||54999e4000df49298418febd, this is a drop 2014-12-23T17:15:02.864+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-21 at shard version 3|3||54999e4000df49298418febd, took 0ms 2014-12-23T17:15:03.951+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-22 2014-12-23T17:15:04.022+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-22 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e4600df49298418fec1, current metadata version is 3|3||54999e4600df49298418fec1 2014-12-23T17:15:04.023+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-22, previous version was 0|0||54999e4600df49298418fec1, this is a drop 2014-12-23T17:15:04.023+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-22 at shard version 3|3||54999e4600df49298418fec1, took 0ms 2014-12-23T17:15:05.086+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-23 2014-12-23T17:15:05.151+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-23 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e4e00df49298418fec5, current metadata version is 3|3||54999e4e00df49298418fec5 2014-12-23T17:15:05.152+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-23, previous version was 0|0||54999e4e00df49298418fec5, this is a drop 2014-12-23T17:15:05.152+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-23 at shard version 3|3||54999e4e00df49298418fec5, took 0ms 2014-12-23T17:15:06.181+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-24 2014-12-23T17:15:06.247+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-24 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e5400df49298418feca, current metadata version is 3|3||54999e5400df49298418feca 2014-12-23T17:15:06.248+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-24, previous version was 0|0||54999e5400df49298418feca, this is a drop 2014-12-23T17:15:06.248+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-24 at shard version 3|3||54999e5400df49298418feca, took 0ms 2014-12-23T17:15:07.360+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-25 2014-12-23T17:15:07.409+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-25 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e5c00df49298418fecd, current metadata version is 3|3||54999e5c00df49298418fecd 2014-12-23T17:15:07.410+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-25, previous version was 0|0||54999e5c00df49298418fecd, this is a drop 2014-12-23T17:15:07.410+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-25 at shard version 3|3||54999e5c00df49298418fecd, took 0ms 2014-12-23T17:15:08.498+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-26 2014-12-23T17:15:08.560+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e6200df49298418fed2, current metadata version is 3|3||54999e6200df49298418fed2 2014-12-23T17:15:08.560+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-26, previous version was 0|0||54999e6200df49298418fed2, this is a drop 2014-12-23T17:15:08.560+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-26 at shard version 3|3||54999e6200df49298418fed2, took 0ms 2014-12-23T17:15:09.629+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-27 2014-12-23T17:15:09.694+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-27 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e6900df49298418fed6, current metadata version is 3|3||54999e6900df49298418fed6 2014-12-23T17:15:09.694+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-27, previous version was 0|0||54999e6900df49298418fed6, this is a drop 2014-12-23T17:15:09.694+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-27 at shard version 3|3||54999e6900df49298418fed6, took 0ms 2014-12-23T17:15:10.717+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-28 2014-12-23T17:15:10.800+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-28 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e7000df49298418feda, current metadata version is 3|3||54999e7000df49298418feda 2014-12-23T17:15:10.801+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-28, previous version was 0|0||54999e7000df49298418feda, this is a drop 2014-12-23T17:15:10.801+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-28 at shard version 3|3||54999e7000df49298418feda, took 0ms 2014-12-23T17:15:11.597+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-29 2014-12-23T17:15:11.626+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e7600df49298418fede, current metadata version is 3|3||54999e7600df49298418fede 2014-12-23T17:15:11.627+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-29, previous version was 0|0||54999e7600df49298418fede, this is a drop 2014-12-23T17:15:11.627+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-29 at shard version 3|3||54999e7600df49298418fede, took 0ms 2014-12-23T17:15:12.257+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-3 2014-12-23T17:15:12.291+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999dc500df49298418fe6f, current metadata version is 3|3||54999dc500df49298418fe6f 2014-12-23T17:15:12.292+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-3, previous version was 0|0||54999dc500df49298418fe6f, this is a drop 2014-12-23T17:15:12.292+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-3 at shard version 3|3||54999dc500df49298418fe6f, took 0ms 2014-12-23T17:15:12.994+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-30 2014-12-23T17:15:13.068+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e7e00df49298418fee2, current metadata version is 3|3||54999e7e00df49298418fee2 2014-12-23T17:15:13.068+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-30, previous version was 0|0||54999e7e00df49298418fee2, this is a drop 2014-12-23T17:15:13.069+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-30 at shard version 3|3||54999e7e00df49298418fee2, took 0ms 2014-12-23T17:15:14.123+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-31 2014-12-23T17:15:14.191+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-31 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e8400df49298418fee7, current metadata version is 3|3||54999e8400df49298418fee7 2014-12-23T17:15:14.192+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-31, previous version was 0|0||54999e8400df49298418fee7, this is a drop 2014-12-23T17:15:14.192+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-31 at shard version 3|3||54999e8400df49298418fee7, took 0ms 2014-12-23T17:15:15.262+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-32 2014-12-23T17:15:15.330+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-32 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e8b00df49298418feea, current metadata version is 3|3||54999e8b00df49298418feea 2014-12-23T17:15:15.330+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-32, previous version was 0|0||54999e8b00df49298418feea, this is a drop 2014-12-23T17:15:15.331+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-32 at shard version 3|3||54999e8b00df49298418feea, took 0ms 2014-12-23T17:15:16.404+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-33 2014-12-23T17:15:16.469+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-33 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e9100df49298418feef, current metadata version is 3|3||54999e9100df49298418feef 2014-12-23T17:15:16.470+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-33, previous version was 0|0||54999e9100df49298418feef, this is a drop 2014-12-23T17:15:16.470+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-33 at shard version 3|3||54999e9100df49298418feef, took 0ms 2014-12-23T17:15:17.540+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-34 2014-12-23T17:15:17.605+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-34 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e9900df49298418fef2, current metadata version is 3|3||54999e9900df49298418fef2 2014-12-23T17:15:17.606+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-34, previous version was 0|0||54999e9900df49298418fef2, this is a drop 2014-12-23T17:15:17.606+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-34 at shard version 3|3||54999e9900df49298418fef2, took 0ms 2014-12-23T17:15:18.678+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-35 2014-12-23T17:15:18.741+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-35 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999e9f00df49298418fef7, current metadata version is 3|3||54999e9f00df49298418fef7 2014-12-23T17:15:18.742+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-35, previous version was 0|0||54999e9f00df49298418fef7, this is a drop 2014-12-23T17:15:18.742+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-35 at shard version 3|3||54999e9f00df49298418fef7, took 0ms 2014-12-23T17:15:19.795+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-36 2014-12-23T17:15:19.857+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-36 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ea600df49298418fefb, current metadata version is 3|3||54999ea600df49298418fefb 2014-12-23T17:15:19.858+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-36, previous version was 0|0||54999ea600df49298418fefb, this is a drop 2014-12-23T17:15:19.858+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-36 at shard version 3|3||54999ea600df49298418fefb, took 0ms 2014-12-23T17:15:20.870+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-37 2014-12-23T17:15:20.949+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-37 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ead00df49298418feff, current metadata version is 3|3||54999ead00df49298418feff 2014-12-23T17:15:20.950+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-37, previous version was 0|0||54999ead00df49298418feff, this is a drop 2014-12-23T17:15:20.950+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-37 at shard version 3|3||54999ead00df49298418feff, took 0ms 2014-12-23T17:15:22.029+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-38 2014-12-23T17:15:22.087+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-38 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999eb300df49298418ff03, current metadata version is 3|3||54999eb300df49298418ff03 2014-12-23T17:15:22.088+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-38, previous version was 0|0||54999eb300df49298418ff03, this is a drop 2014-12-23T17:15:22.088+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-38 at shard version 3|3||54999eb300df49298418ff03, took 0ms 2014-12-23T17:15:23.162+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-39 2014-12-23T17:15:23.219+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-39 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999eba00df49298418ff07, current metadata version is 3|3||54999eba00df49298418ff07 2014-12-23T17:15:23.220+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-39, previous version was 0|0||54999eba00df49298418ff07, this is a drop 2014-12-23T17:15:23.220+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-39 at shard version 3|3||54999eba00df49298418ff07, took 0ms 2014-12-23T17:15:24.025+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-4 2014-12-23T17:15:24.083+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:15:23 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:15:24.087+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999dcc00df49298418fe74, current metadata version is 3|3||54999dcc00df49298418fe74 2014-12-23T17:15:24.088+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-4, previous version was 0|0||54999dcc00df49298418fe74, this is a drop 2014-12-23T17:15:24.088+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-4 at shard version 3|3||54999dcc00df49298418fe74, took 0ms 2014-12-23T17:15:24.734+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-40 2014-12-23T17:15:24.762+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-40 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ec000df49298418ff0c, current metadata version is 3|3||54999ec000df49298418ff0c 2014-12-23T17:15:24.763+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-40, previous version was 0|0||54999ec000df49298418ff0c, this is a drop 2014-12-23T17:15:24.763+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-40 at shard version 3|3||54999ec000df49298418ff0c, took 0ms 2014-12-23T17:15:25.663+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-41 2014-12-23T17:15:25.731+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-41 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ec800df49298418ff0f, current metadata version is 3|3||54999ec800df49298418ff0f 2014-12-23T17:15:25.732+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-41, previous version was 0|0||54999ec800df49298418ff0f, this is a drop 2014-12-23T17:15:25.732+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-41 at shard version 3|3||54999ec800df49298418ff0f, took 0ms 2014-12-23T17:15:26.748+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-42 2014-12-23T17:15:26.811+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-42 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ece00df49298418ff14, current metadata version is 3|3||54999ece00df49298418ff14 2014-12-23T17:15:26.812+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-42, previous version was 0|0||54999ece00df49298418ff14, this is a drop 2014-12-23T17:15:26.812+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-42 at shard version 3|3||54999ece00df49298418ff14, took 0ms 2014-12-23T17:15:27.894+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-43 2014-12-23T17:15:27.956+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-43 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ed500df49298418ff17, current metadata version is 3|3||54999ed500df49298418ff17 2014-12-23T17:15:27.957+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-43, previous version was 0|0||54999ed500df49298418ff17, this is a drop 2014-12-23T17:15:27.957+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-43 at shard version 3|3||54999ed500df49298418ff17, took 0ms 2014-12-23T17:15:29.014+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-44 2014-12-23T17:15:29.076+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-44 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999edb00df49298418ff1c, current metadata version is 3|3||54999edb00df49298418ff1c 2014-12-23T17:15:29.077+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-44, previous version was 0|0||54999edb00df49298418ff1c, this is a drop 2014-12-23T17:15:29.077+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-44 at shard version 3|3||54999edb00df49298418ff1c, took 0ms 2014-12-23T17:15:30.148+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-45 2014-12-23T17:15:30.235+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-45 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ee300df49298418ff1f, current metadata version is 3|3||54999ee300df49298418ff1f 2014-12-23T17:15:30.235+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-45, previous version was 0|0||54999ee300df49298418ff1f, this is a drop 2014-12-23T17:15:30.235+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-45 at shard version 3|3||54999ee300df49298418ff1f, took 0ms 2014-12-23T17:15:31.296+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-46 2014-12-23T17:15:31.372+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-46 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ee900df49298418ff24, current metadata version is 3|3||54999ee900df49298418ff24 2014-12-23T17:15:31.373+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-46, previous version was 0|0||54999ee900df49298418ff24, this is a drop 2014-12-23T17:15:31.373+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-46 at shard version 3|3||54999ee900df49298418ff24, took 0ms 2014-12-23T17:15:32.442+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-47 2014-12-23T17:15:32.505+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-47 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ef100df49298418ff28, current metadata version is 3|3||54999ef100df49298418ff28 2014-12-23T17:15:32.506+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-47, previous version was 0|0||54999ef100df49298418ff28, this is a drop 2014-12-23T17:15:32.506+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-47 at shard version 3|3||54999ef100df49298418ff28, took 0ms 2014-12-23T17:15:33.561+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-48 2014-12-23T17:15:33.623+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-48 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ef700df49298418ff2c, current metadata version is 3|3||54999ef700df49298418ff2c 2014-12-23T17:15:33.623+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-48, previous version was 0|0||54999ef700df49298418ff2c, this is a drop 2014-12-23T17:15:33.623+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-48 at shard version 3|3||54999ef700df49298418ff2c, took 0ms 2014-12-23T17:15:34.693+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-49 2014-12-23T17:15:34.751+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-49 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999efe00df49298418ff30, current metadata version is 3|3||54999efe00df49298418ff30 2014-12-23T17:15:34.752+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-49, previous version was 0|0||54999efe00df49298418ff30, this is a drop 2014-12-23T17:15:34.752+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-49 at shard version 3|3||54999efe00df49298418ff30, took 0ms 2014-12-23T17:15:35.668+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-5 2014-12-23T17:15:35.746+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999dd300df49298418fe79, current metadata version is 3|3||54999dd300df49298418fe79 2014-12-23T17:15:35.747+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-5, previous version was 0|0||54999dd300df49298418fe79, this is a drop 2014-12-23T17:15:35.747+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-5 at shard version 3|3||54999dd300df49298418fe79, took 0ms 2014-12-23T17:15:36.435+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-50 2014-12-23T17:15:36.465+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-50 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f0400df49298418ff34, current metadata version is 3|3||54999f0400df49298418ff34 2014-12-23T17:15:36.466+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-50, previous version was 0|0||54999f0400df49298418ff34, this is a drop 2014-12-23T17:15:36.466+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-50 at shard version 3|3||54999f0400df49298418ff34, took 0ms 2014-12-23T17:15:37.419+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-51 2014-12-23T17:15:37.480+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-51 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f0b00df49298418ff38, current metadata version is 3|3||54999f0b00df49298418ff38 2014-12-23T17:15:37.481+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-51, previous version was 0|0||54999f0b00df49298418ff38, this is a drop 2014-12-23T17:15:37.481+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-51 at shard version 3|3||54999f0b00df49298418ff38, took 0ms 2014-12-23T17:15:38.562+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-52 2014-12-23T17:15:38.625+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f1100df49298418ff3c, current metadata version is 3|3||54999f1100df49298418ff3c 2014-12-23T17:15:38.625+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-52, previous version was 0|0||54999f1100df49298418ff3c, this is a drop 2014-12-23T17:15:38.625+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-52 at shard version 3|3||54999f1100df49298418ff3c, took 0ms 2014-12-23T17:15:39.680+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-53 2014-12-23T17:15:39.744+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-53 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f1800df49298418ff40, current metadata version is 3|3||54999f1800df49298418ff40 2014-12-23T17:15:39.744+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-53, previous version was 0|0||54999f1800df49298418ff40, this is a drop 2014-12-23T17:15:39.744+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-53 at shard version 3|3||54999f1800df49298418ff40, took 0ms 2014-12-23T17:15:40.812+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-54 2014-12-23T17:15:40.876+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-54 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f1f00df49298418ff44, current metadata version is 3|3||54999f1f00df49298418ff44 2014-12-23T17:15:40.876+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-54, previous version was 0|0||54999f1f00df49298418ff44, this is a drop 2014-12-23T17:15:40.876+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-54 at shard version 3|3||54999f1f00df49298418ff44, took 0ms 2014-12-23T17:15:41.930+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-55 2014-12-23T17:15:41.985+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-55 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f2500df49298418ff48, current metadata version is 3|3||54999f2500df49298418ff48 2014-12-23T17:15:41.986+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-55, previous version was 0|0||54999f2500df49298418ff48, this is a drop 2014-12-23T17:15:41.986+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-55 at shard version 3|3||54999f2500df49298418ff48, took 0ms 2014-12-23T17:15:43.052+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-56 2014-12-23T17:15:43.112+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-56 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f2c00df49298418ff4c, current metadata version is 3|3||54999f2c00df49298418ff4c 2014-12-23T17:15:43.112+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-56, previous version was 0|0||54999f2c00df49298418ff4c, this is a drop 2014-12-23T17:15:43.112+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-56 at shard version 3|3||54999f2c00df49298418ff4c, took 0ms 2014-12-23T17:15:44.168+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-57 2014-12-23T17:15:44.230+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-57 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f3300df49298418ff50, current metadata version is 3|3||54999f3300df49298418ff50 2014-12-23T17:15:44.231+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-57, previous version was 0|0||54999f3300df49298418ff50, this is a drop 2014-12-23T17:15:44.231+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-57 at shard version 3|3||54999f3300df49298418ff50, took 0ms 2014-12-23T17:15:45.297+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-58 2014-12-23T17:15:45.367+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-58 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f3900df49298418ff54, current metadata version is 3|3||54999f3900df49298418ff54 2014-12-23T17:15:45.368+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-58, previous version was 0|0||54999f3900df49298418ff54, this is a drop 2014-12-23T17:15:45.368+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-58 at shard version 3|3||54999f3900df49298418ff54, took 0ms 2014-12-23T17:15:46.433+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-59 2014-12-23T17:15:46.494+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-59 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f4000df49298418ff58, current metadata version is 3|3||54999f4000df49298418ff58 2014-12-23T17:15:46.494+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-59, previous version was 0|0||54999f4000df49298418ff58, this is a drop 2014-12-23T17:15:46.494+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-59 at shard version 3|3||54999f4000df49298418ff58, took 0ms 2014-12-23T17:15:47.402+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-6 2014-12-23T17:15:47.433+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 with requested shard version 0|0||000000000000000000000000, current shard version is 3|0||54999dda00df49298418fe7c, current metadata version is 3|0||54999dda00df49298418fe7c 2014-12-23T17:15:47.434+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-6, previous version was 0|0||54999dda00df49298418fe7c, this is a drop 2014-12-23T17:15:47.434+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-6 at shard version 3|0||54999dda00df49298418fe7c, took 0ms 2014-12-23T17:15:48.370+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-60 2014-12-23T17:15:48.438+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-60 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f4600df49298418ff5c, current metadata version is 3|3||54999f4600df49298418ff5c 2014-12-23T17:15:48.439+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-60, previous version was 0|0||54999f4600df49298418ff5c, this is a drop 2014-12-23T17:15:48.439+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-60 at shard version 3|3||54999f4600df49298418ff5c, took 0ms 2014-12-23T17:15:49.496+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-61 2014-12-23T17:15:49.562+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-61 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f4d00df49298418ff60, current metadata version is 3|3||54999f4d00df49298418ff60 2014-12-23T17:15:49.563+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-61, previous version was 0|0||54999f4d00df49298418ff60, this is a drop 2014-12-23T17:15:49.563+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-61 at shard version 3|3||54999f4d00df49298418ff60, took 0ms 2014-12-23T17:15:50.623+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-62 2014-12-23T17:15:50.686+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f5300df49298418ff64, current metadata version is 3|3||54999f5300df49298418ff64 2014-12-23T17:15:50.687+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-62, previous version was 0|0||54999f5300df49298418ff64, this is a drop 2014-12-23T17:15:50.687+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-62 at shard version 3|3||54999f5300df49298418ff64, took 0ms 2014-12-23T17:15:51.764+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-63 2014-12-23T17:15:51.831+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999f5a00df49298418ff68, current metadata version is 3|3||54999f5a00df49298418ff68 2014-12-23T17:15:51.832+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-63, previous version was 0|0||54999f5a00df49298418ff68, this is a drop 2014-12-23T17:15:51.832+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-63 at shard version 3|3||54999f5a00df49298418ff68, took 0ms 2014-12-23T17:15:52.884+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-7 2014-12-23T17:15:52.942+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999ddc00df49298418fe80, current metadata version is 3|3||54999ddc00df49298418fe80 2014-12-23T17:15:52.942+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-7, previous version was 0|0||54999ddc00df49298418fe80, this is a drop 2014-12-23T17:15:52.942+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-7 at shard version 3|3||54999ddc00df49298418fe80, took 0ms 2014-12-23T17:15:54.012+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-8 2014-12-23T17:15:54.070+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999de400df49298418fe86, current metadata version is 3|3||54999de400df49298418fe86 2014-12-23T17:15:54.071+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-8, previous version was 0|0||54999de400df49298418fe86, this is a drop 2014-12-23T17:15:54.071+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-8 at shard version 3|3||54999de400df49298418fe86, took 0ms 2014-12-23T17:15:54.581+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:15:54 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:15:55.142+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-9 2014-12-23T17:15:55.206+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-9 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||54999deb00df49298418fe8b, current metadata version is 3|3||54999deb00df49298418fe8b 2014-12-23T17:15:55.206+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-9, previous version was 0|0||54999deb00df49298418fe8b, this is a drop 2014-12-23T17:15:55.207+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-9 at shard version 3|3||54999deb00df49298418fe8b, took 0ms 2014-12-23T17:15:55.606+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:41443 #6 (6 connections now open) 2014-12-23T17:15:55.606+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:15:55.606+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:15:55.700+0000 I INDEX [conn2] build index on: benchdb1.COL-0 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-0" } 2014-12-23T17:15:55.700+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:15:55.700+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:15:55.952+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 1|2||5499a34800df4929841900ce, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:15:55.953+0000 I SHARDING [conn2] collection benchdb1.COL-0 was previously unsharded, new metadata loaded with shard version 1|2||5499a34800df4929841900ce 2014-12-23T17:15:55.953+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a34800df4929841900ce, took 0ms 2014-12-23T17:15:55.993+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:15:55.994+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34800df4929841900ce') } 2014-12-23T17:15:56.555+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a34b9a8f659f34bfb4f7 2014-12-23T17:15:56.555+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:15:56-5499a34c9a8f659f34bfb4f8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354956555), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:15:56.622+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 1|2||5499a34800df4929841900ce, current metadata version is 1|2||5499a34800df4929841900ce 2014-12-23T17:15:56.623+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-0 already up to date (shard version : 1|2||5499a34800df4929841900ce, took 0ms) 2014-12-23T17:15:56.623+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a34800df4929841900ce 2014-12-23T17:15:56.623+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:15:56.626+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:56.628+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:56.633+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:56.633+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:15:56.633+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:15:56.633+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a34800df4929841900ce 2014-12-23T17:15:56.687+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:15:56.687+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a34800df4929841900ce through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:15:56.992+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:15:56-5499a34c9a8f659f34bfb4f9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354956992), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:15:57.053+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:15:57.053+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:15:57.053+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2236 2014-12-23T17:15:57.053+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:15:57.053+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:15:57.356+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:15:57.356+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:15:57-5499a34d9a8f659f34bfb4fa", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354957356), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 629, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 420, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:15:57.405+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34800df4929841900ce') } ntoreturn:1 keyUpdates:0 reslen:37 1412ms 2014-12-23T17:15:57.407+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:15:57.407+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34800df4929841900ce') } 2014-12-23T17:15:58.014+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a34d9a8f659f34bfb4fb 2014-12-23T17:15:58.014+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:15:58-5499a34e9a8f659f34bfb4fc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354958014), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:15:58.085+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 2|0||5499a34800df4929841900ce, current metadata version is 2|0||5499a34800df4929841900ce 2014-12-23T17:15:58.086+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 2|0||5499a34800df4929841900ce to shard version 2|1||5499a34800df4929841900ce 2014-12-23T17:15:58.086+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a34800df4929841900ce, took 0ms 2014-12-23T17:15:58.086+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a34800df4929841900ce 2014-12-23T17:15:58.086+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:15:58.088+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:58.091+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:58.095+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:58.104+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:58.120+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:15:58.120+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:15:58.120+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:15:58.120+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a34800df4929841900ce 2014-12-23T17:15:58.183+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:15:58.183+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a34800df4929841900ce through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:15:58.489+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:15:58-5499a34e9a8f659f34bfb4fd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354958489), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:15:58.544+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:15:58.544+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:15:58.544+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2244 2014-12-23T17:15:58.544+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:15:58.544+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:15:58.809+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:15:58.809+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:15:58-5499a34e9a8f659f34bfb4fe", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354958809), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 678, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:15:58.863+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34800df4929841900ce') } ntoreturn:1 keyUpdates:0 reslen:37 1456ms 2014-12-23T17:15:58.865+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a34800df4929841900ce') } 2014-12-23T17:15:59.471+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a34e9a8f659f34bfb4ff 2014-12-23T17:15:59.471+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 3|0||5499a34800df4929841900ce, current metadata version is 3|0||5499a34800df4929841900ce 2014-12-23T17:15:59.472+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 3|0||5499a34800df4929841900ce to shard version 3|1||5499a34800df4929841900ce 2014-12-23T17:15:59.472+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a34800df4929841900ce, took 0ms 2014-12-23T17:15:59.472+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a34800df4929841900ce 2014-12-23T17:15:59.757+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:15:59-5499a34f9a8f659f34bfb500", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354959757), what: "split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a34800df4929841900ce') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a34800df4929841900ce') } } } 2014-12-23T17:16:00.137+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:00.137+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a34800df4929841900ce') } ntoreturn:1 keyUpdates:0 reslen:37 1272ms 2014-12-23T17:16:02.583+0000 I INDEX [conn2] build index on: benchdb1.COL-1 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-1" } 2014-12-23T17:16:02.583+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:02.584+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:02.879+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 1|2||5499a34f00df4929841900d3, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:02.880+0000 I SHARDING [conn2] collection benchdb1.COL-1 was previously unsharded, new metadata loaded with shard version 1|2||5499a34f00df4929841900d3 2014-12-23T17:16:02.880+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a34f00df4929841900d3, took 0ms 2014-12-23T17:16:02.919+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:02.919+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34f00df4929841900d3') } 2014-12-23T17:16:03.528+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3529a8f659f34bfb502 2014-12-23T17:16:03.528+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:03-5499a3539a8f659f34bfb503", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354963528), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:03.589+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||5499a34f00df4929841900d3, current metadata version is 1|2||5499a34f00df4929841900d3 2014-12-23T17:16:03.590+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||5499a34f00df4929841900d3, took 0ms) 2014-12-23T17:16:03.590+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a34f00df4929841900d3 2014-12-23T17:16:03.591+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:03.594+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:03.596+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:03.601+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:03.601+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:03.601+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:03.601+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a34f00df4929841900d3 2014-12-23T17:16:03.652+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:03.652+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a34f00df4929841900d3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:16:03.958+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:03-5499a3539a8f659f34bfb504", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354963958), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:04.020+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:04.020+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:04.020+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2265 2014-12-23T17:16:04.020+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:04.020+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:04.324+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:04.324+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:04-5499a3549a8f659f34bfb505", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354964324), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 419, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:04.384+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34f00df4929841900d3') } ntoreturn:1 keyUpdates:0 reslen:37 1465ms 2014-12-23T17:16:04.386+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:04.386+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34f00df4929841900d3') } 2014-12-23T17:16:04.994+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3549a8f659f34bfb506 2014-12-23T17:16:04.994+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:04-5499a3549a8f659f34bfb507", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354964994), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:05.051+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 2|0||5499a34f00df4929841900d3, current metadata version is 2|0||5499a34f00df4929841900d3 2014-12-23T17:16:05.052+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-1 from shard version 2|0||5499a34f00df4929841900d3 to shard version 2|1||5499a34f00df4929841900d3 2014-12-23T17:16:05.052+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a34f00df4929841900d3, took 0ms 2014-12-23T17:16:05.052+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a34f00df4929841900d3 2014-12-23T17:16:05.052+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:05.055+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:05.057+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:05.062+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:05.070+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:05.087+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:05.087+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:05.087+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:05.087+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a34f00df4929841900d3 2014-12-23T17:16:05.142+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:05.142+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a34f00df4929841900d3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:16:05.448+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:05-5499a3559a8f659f34bfb508", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354965448), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:05.533+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:05.533+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:05.533+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2273 2014-12-23T17:16:05.533+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:05.533+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:05.837+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:05.837+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:05-5499a3559a8f659f34bfb509", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354965837), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 665, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 445, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:05.890+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a34f00df4929841900d3') } ntoreturn:1 keyUpdates:0 reslen:37 1504ms 2014-12-23T17:16:05.892+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a34f00df4929841900d3') } 2014-12-23T17:16:06.500+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3559a8f659f34bfb50a 2014-12-23T17:16:06.500+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 3|0||5499a34f00df4929841900d3, current metadata version is 3|0||5499a34f00df4929841900d3 2014-12-23T17:16:06.501+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-1 from shard version 3|0||5499a34f00df4929841900d3 to shard version 3|1||5499a34f00df4929841900d3 2014-12-23T17:16:06.501+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a34f00df4929841900d3, took 0ms 2014-12-23T17:16:06.501+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a34f00df4929841900d3 2014-12-23T17:16:06.806+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:06-5499a3569a8f659f34bfb50b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354966806), what: "split", ns: "benchdb1.COL-1", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a34f00df4929841900d3') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a34f00df4929841900d3') } } } 2014-12-23T17:16:07.180+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:07.180+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a34f00df4929841900d3') } ntoreturn:1 keyUpdates:0 reslen:37 1288ms 2014-12-23T17:16:09.661+0000 I INDEX [conn2] build index on: benchdb1.COL-10 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-10" } 2014-12-23T17:16:09.661+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:09.662+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:09.954+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 1|2||5499a35600df4929841900d8, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:09.955+0000 I SHARDING [conn2] collection benchdb1.COL-10 was previously unsharded, new metadata loaded with shard version 1|2||5499a35600df4929841900d8 2014-12-23T17:16:09.955+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a35600df4929841900d8, took 0ms 2014-12-23T17:16:09.996+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:09.997+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35600df4929841900d8') } 2014-12-23T17:16:10.604+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3599a8f659f34bfb50d 2014-12-23T17:16:10.604+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:10-5499a35a9a8f659f34bfb50e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354970604), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:10.661+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 1|2||5499a35600df4929841900d8, current metadata version is 1|2||5499a35600df4929841900d8 2014-12-23T17:16:10.662+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-10 already up to date (shard version : 1|2||5499a35600df4929841900d8, took 0ms) 2014-12-23T17:16:10.662+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a35600df4929841900d8 2014-12-23T17:16:10.662+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:10.665+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:10.667+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:10.672+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:10.672+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:10.672+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:10.672+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a35600df4929841900d8 2014-12-23T17:16:10.725+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:10.725+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a35600df4929841900d8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:16:11.030+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:11-5499a35b9a8f659f34bfb50f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354971030), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:11.086+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:11.086+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:11.086+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2294 2014-12-23T17:16:11.086+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:11.086+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:11.389+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:11.389+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:11-5499a35b9a8f659f34bfb510", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354971389), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 665, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:11.452+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35600df4929841900d8') } ntoreturn:1 keyUpdates:0 reslen:37 1455ms 2014-12-23T17:16:11.453+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:11.454+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35600df4929841900d8') } 2014-12-23T17:16:12.062+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a35b9a8f659f34bfb511 2014-12-23T17:16:12.062+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:12-5499a35c9a8f659f34bfb512", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354972062), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:12.118+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 2|0||5499a35600df4929841900d8, current metadata version is 2|0||5499a35600df4929841900d8 2014-12-23T17:16:12.119+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-10 from shard version 2|0||5499a35600df4929841900d8 to shard version 2|1||5499a35600df4929841900d8 2014-12-23T17:16:12.119+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a35600df4929841900d8, took 0ms 2014-12-23T17:16:12.119+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a35600df4929841900d8 2014-12-23T17:16:12.119+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:12.121+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:12.124+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:12.128+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:12.137+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:12.153+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:12.153+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:12.153+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:12.153+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a35600df4929841900d8 2014-12-23T17:16:12.213+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:12.213+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a35600df4929841900d8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:16:12.518+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:12-5499a35c9a8f659f34bfb513", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354972518), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:12.569+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:12.569+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:12.569+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2302 2014-12-23T17:16:12.569+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:12.569+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:12.873+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:12.873+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:12-5499a35c9a8f659f34bfb514", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354972873), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 665, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 416, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:12.937+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35600df4929841900d8') } ntoreturn:1 keyUpdates:0 reslen:37 1484ms 2014-12-23T17:16:12.939+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a35600df4929841900d8') } 2014-12-23T17:16:13.546+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a35c9a8f659f34bfb515 2014-12-23T17:16:13.546+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 3|0||5499a35600df4929841900d8, current metadata version is 3|0||5499a35600df4929841900d8 2014-12-23T17:16:13.547+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-10 from shard version 3|0||5499a35600df4929841900d8 to shard version 3|1||5499a35600df4929841900d8 2014-12-23T17:16:13.547+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a35600df4929841900d8, took 0ms 2014-12-23T17:16:13.547+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a35600df4929841900d8 2014-12-23T17:16:13.852+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:13-5499a35d9a8f659f34bfb516", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354973852), what: "split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a35600df4929841900d8') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a35600df4929841900d8') } } } 2014-12-23T17:16:14.223+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:14.223+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a35600df4929841900d8') } ntoreturn:1 keyUpdates:0 reslen:37 1283ms 2014-12-23T17:16:16.803+0000 I INDEX [conn2] build index on: benchdb1.COL-11 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-11" } 2014-12-23T17:16:16.803+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:16.804+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:17.074+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 1|2||5499a35d00df4929841900db, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:17.074+0000 I SHARDING [conn2] collection benchdb1.COL-11 was previously unsharded, new metadata loaded with shard version 1|2||5499a35d00df4929841900db 2014-12-23T17:16:17.075+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a35d00df4929841900db, took 0ms 2014-12-23T17:16:17.113+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:17.113+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35d00df4929841900db') } 2014-12-23T17:16:17.720+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3619a8f659f34bfb518 2014-12-23T17:16:17.720+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:17-5499a3619a8f659f34bfb519", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354977720), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:17.780+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 1|2||5499a35d00df4929841900db, current metadata version is 1|2||5499a35d00df4929841900db 2014-12-23T17:16:17.781+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-11 already up to date (shard version : 1|2||5499a35d00df4929841900db, took 0ms) 2014-12-23T17:16:17.781+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a35d00df4929841900db 2014-12-23T17:16:17.782+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:17.784+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:17.787+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:17.791+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:17.791+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:17.791+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:17.791+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a35d00df4929841900db 2014-12-23T17:16:17.847+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:17.848+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a35d00df4929841900db through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T17:16:18.152+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:18-5499a3629a8f659f34bfb51a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354978152), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:18.215+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:18.215+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:18.215+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2320 2014-12-23T17:16:18.215+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:18.215+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:18.518+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:18.518+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:18-5499a3629a8f659f34bfb51b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354978518), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 423, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:18.607+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35d00df4929841900db') } ntoreturn:1 keyUpdates:0 reslen:37 1493ms 2014-12-23T17:16:18.608+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:18.609+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35d00df4929841900db') } 2014-12-23T17:16:19.215+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3629a8f659f34bfb51c 2014-12-23T17:16:19.215+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:19-5499a3639a8f659f34bfb51d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354979215), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:19.270+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 2|0||5499a35d00df4929841900db, current metadata version is 2|0||5499a35d00df4929841900db 2014-12-23T17:16:19.271+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-11 from shard version 2|0||5499a35d00df4929841900db to shard version 2|1||5499a35d00df4929841900db 2014-12-23T17:16:19.271+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a35d00df4929841900db, took 0ms 2014-12-23T17:16:19.271+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a35d00df4929841900db 2014-12-23T17:16:19.272+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:19.274+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:19.276+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:19.281+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:19.289+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:19.305+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:19.306+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:19.306+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:19.306+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a35d00df4929841900db 2014-12-23T17:16:19.362+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:19.362+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a35d00df4929841900db through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T17:16:19.666+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:19-5499a3639a8f659f34bfb51e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354979666), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:19.716+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:19.716+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:19.716+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2328 2014-12-23T17:16:19.716+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:19.716+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:19.924+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:19.924+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:19-5499a3639a8f659f34bfb51f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354979924), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 410, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:19.987+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a35d00df4929841900db') } ntoreturn:1 keyUpdates:0 reslen:37 1378ms 2014-12-23T17:16:19.988+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a35d00df4929841900db') } 2014-12-23T17:16:20.482+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3639a8f659f34bfb520 2014-12-23T17:16:20.482+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 3|0||5499a35d00df4929841900db, current metadata version is 3|0||5499a35d00df4929841900db 2014-12-23T17:16:20.483+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-11 from shard version 3|0||5499a35d00df4929841900db to shard version 3|1||5499a35d00df4929841900db 2014-12-23T17:16:20.483+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a35d00df4929841900db, took 0ms 2014-12-23T17:16:20.483+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a35d00df4929841900db 2014-12-23T17:16:20.715+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:20-5499a3649a8f659f34bfb521", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354980715), what: "split", ns: "benchdb1.COL-11", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a35d00df4929841900db') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a35d00df4929841900db') } } } 2014-12-23T17:16:21.089+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:21.089+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a35d00df4929841900db') } ntoreturn:1 keyUpdates:0 reslen:37 1100ms 2014-12-23T17:16:23.672+0000 I INDEX [conn2] build index on: benchdb1.COL-12 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-12" } 2014-12-23T17:16:23.672+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:23.672+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:23.953+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 1|2||5499a36400df4929841900e0, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:23.954+0000 I SHARDING [conn2] collection benchdb1.COL-12 was previously unsharded, new metadata loaded with shard version 1|2||5499a36400df4929841900e0 2014-12-23T17:16:23.954+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a36400df4929841900e0, took 0ms 2014-12-23T17:16:23.994+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:23.995+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36400df4929841900e0') } 2014-12-23T17:16:24.602+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3679a8f659f34bfb523 2014-12-23T17:16:24.602+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:24-5499a3689a8f659f34bfb524", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354984602), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:24.660+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 1|2||5499a36400df4929841900e0, current metadata version is 1|2||5499a36400df4929841900e0 2014-12-23T17:16:24.661+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-12 already up to date (shard version : 1|2||5499a36400df4929841900e0, took 0ms) 2014-12-23T17:16:24.661+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a36400df4929841900e0 2014-12-23T17:16:24.661+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:24.664+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:24.666+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:24.671+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:24.671+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:24.671+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:24.671+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a36400df4929841900e0 2014-12-23T17:16:24.730+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:24.731+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a36400df4929841900e0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T17:16:25.034+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:16:24 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:16:25.035+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:25-5499a3699a8f659f34bfb525", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354985035), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:25.091+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:25.091+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:25.091+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-12 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2349 2014-12-23T17:16:25.092+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:25.092+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:25.394+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:25.395+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:25-5499a3699a8f659f34bfb526", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354985395), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 665, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 420, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:25.449+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36400df4929841900e0') } ntoreturn:1 keyUpdates:0 reslen:37 1454ms 2014-12-23T17:16:25.450+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:25.451+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36400df4929841900e0') } 2014-12-23T17:16:26.057+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3699a8f659f34bfb527 2014-12-23T17:16:26.057+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:26-5499a36a9a8f659f34bfb528", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354986057), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:26.117+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 2|0||5499a36400df4929841900e0, current metadata version is 2|0||5499a36400df4929841900e0 2014-12-23T17:16:26.118+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-12 from shard version 2|0||5499a36400df4929841900e0 to shard version 2|1||5499a36400df4929841900e0 2014-12-23T17:16:26.118+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a36400df4929841900e0, took 0ms 2014-12-23T17:16:26.119+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a36400df4929841900e0 2014-12-23T17:16:26.119+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:26.121+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:26.123+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:26.128+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:26.136+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:26.153+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:26.153+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:26.153+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:26.153+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a36400df4929841900e0 2014-12-23T17:16:26.223+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:26.223+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a36400df4929841900e0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T17:16:26.528+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:26-5499a36a9a8f659f34bfb529", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354986528), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:26.578+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:26.578+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:26.578+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2357 2014-12-23T17:16:26.579+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:26.579+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:26.833+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:26.833+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:26-5499a36a9a8f659f34bfb52a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354986833), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 667, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 425, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:26.900+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36400df4929841900e0') } ntoreturn:1 keyUpdates:0 reslen:37 1449ms 2014-12-23T17:16:26.902+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a36400df4929841900e0') } 2014-12-23T17:16:27.507+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a36a9a8f659f34bfb52b 2014-12-23T17:16:27.507+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 3|0||5499a36400df4929841900e0, current metadata version is 3|0||5499a36400df4929841900e0 2014-12-23T17:16:27.508+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-12 from shard version 3|0||5499a36400df4929841900e0 to shard version 3|1||5499a36400df4929841900e0 2014-12-23T17:16:27.508+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a36400df4929841900e0, took 0ms 2014-12-23T17:16:27.508+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a36400df4929841900e0 2014-12-23T17:16:27.813+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:27-5499a36b9a8f659f34bfb52c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354987813), what: "split", ns: "benchdb1.COL-12", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a36400df4929841900e0') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a36400df4929841900e0') } } } 2014-12-23T17:16:28.184+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:28.184+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a36400df4929841900e0') } ntoreturn:1 keyUpdates:0 reslen:37 1282ms 2014-12-23T17:16:30.724+0000 I INDEX [conn2] build index on: benchdb1.COL-13 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-13" } 2014-12-23T17:16:30.724+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:30.724+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:31.042+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 1|2||5499a36b00df4929841900e4, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:31.043+0000 I SHARDING [conn2] collection benchdb1.COL-13 was previously unsharded, new metadata loaded with shard version 1|2||5499a36b00df4929841900e4 2014-12-23T17:16:31.043+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a36b00df4929841900e4, took 0ms 2014-12-23T17:16:31.123+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:31.124+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36b00df4929841900e4') } 2014-12-23T17:16:31.356+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a36b00df4929841900e4') } 2014-12-23T17:16:31.357+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-13 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-13 is taken. 2014-12-23T17:16:31.357+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:31-5499a36f9a8f659f34bfb52f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419354991357), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-13 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKe..." } } 2014-12-23T17:16:31.387+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a36f9a8f659f34bfb52e 2014-12-23T17:16:31.387+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:31-5499a36f9a8f659f34bfb530", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354991387), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:31.460+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 1|2||5499a36b00df4929841900e4, current metadata version is 1|2||5499a36b00df4929841900e4 2014-12-23T17:16:31.461+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-13 already up to date (shard version : 1|2||5499a36b00df4929841900e4, took 0ms) 2014-12-23T17:16:31.461+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a36b00df4929841900e4 2014-12-23T17:16:31.462+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:31.464+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:31.467+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:31.471+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:31.472+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:31.472+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:31.472+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a36b00df4929841900e4 2014-12-23T17:16:31.528+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:31.528+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a36b00df4929841900e4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T17:16:31.833+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:31-5499a36f9a8f659f34bfb531", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354991833), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:31.906+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:31.906+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:31.906+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2376 2014-12-23T17:16:31.906+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:31.906+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:32.164+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:32.164+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:32-5499a3709a8f659f34bfb532", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354992164), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 337, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 434, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:32.220+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36b00df4929841900e4') } ntoreturn:1 keyUpdates:0 reslen:37 1096ms 2014-12-23T17:16:32.222+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:32.222+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36b00df4929841900e4') } 2014-12-23T17:16:32.780+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3709a8f659f34bfb533 2014-12-23T17:16:32.780+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:32-5499a3709a8f659f34bfb534", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354992780), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:32.839+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 2|0||5499a36b00df4929841900e4, current metadata version is 2|0||5499a36b00df4929841900e4 2014-12-23T17:16:32.840+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-13 from shard version 2|0||5499a36b00df4929841900e4 to shard version 2|1||5499a36b00df4929841900e4 2014-12-23T17:16:32.840+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a36b00df4929841900e4, took 0ms 2014-12-23T17:16:32.840+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a36b00df4929841900e4 2014-12-23T17:16:32.841+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:32.843+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:32.845+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:32.850+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:32.858+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:32.875+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:32.875+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:32.875+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:32.875+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a36b00df4929841900e4 2014-12-23T17:16:32.930+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:32.930+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a36b00df4929841900e4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T17:16:33.235+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:33-5499a3719a8f659f34bfb535", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354993235), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:33.285+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:33.285+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:33.286+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2386 2014-12-23T17:16:33.286+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:33.286+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:33.558+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:33.559+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:33-5499a3719a8f659f34bfb536", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354993559), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 618, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 410, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:33.616+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a36b00df4929841900e4') } ntoreturn:1 keyUpdates:0 reslen:37 1394ms 2014-12-23T17:16:33.618+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a36b00df4929841900e4') } 2014-12-23T17:16:34.225+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3719a8f659f34bfb537 2014-12-23T17:16:34.225+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 3|0||5499a36b00df4929841900e4, current metadata version is 3|0||5499a36b00df4929841900e4 2014-12-23T17:16:34.226+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-13 from shard version 3|0||5499a36b00df4929841900e4 to shard version 3|1||5499a36b00df4929841900e4 2014-12-23T17:16:34.226+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a36b00df4929841900e4, took 0ms 2014-12-23T17:16:34.226+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a36b00df4929841900e4 2014-12-23T17:16:34.530+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:34-5499a3729a8f659f34bfb538", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354994530), what: "split", ns: "benchdb1.COL-13", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a36b00df4929841900e4') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a36b00df4929841900e4') } } } 2014-12-23T17:16:34.902+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:34.902+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a36b00df4929841900e4') } ntoreturn:1 keyUpdates:0 reslen:37 1283ms 2014-12-23T17:16:37.483+0000 I INDEX [conn2] build index on: benchdb1.COL-14 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-14" } 2014-12-23T17:16:37.483+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:37.484+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:37.755+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 1|2||5499a37200df4929841900e8, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:37.756+0000 I SHARDING [conn2] collection benchdb1.COL-14 was previously unsharded, new metadata loaded with shard version 1|2||5499a37200df4929841900e8 2014-12-23T17:16:37.756+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a37200df4929841900e8, took 0ms 2014-12-23T17:16:37.798+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:37.798+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37200df4929841900e8') } 2014-12-23T17:16:38.404+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3759a8f659f34bfb53a 2014-12-23T17:16:38.404+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:38-5499a3769a8f659f34bfb53b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354998404), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:38.460+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 1|2||5499a37200df4929841900e8, current metadata version is 1|2||5499a37200df4929841900e8 2014-12-23T17:16:38.461+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-14 already up to date (shard version : 1|2||5499a37200df4929841900e8, took 0ms) 2014-12-23T17:16:38.461+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a37200df4929841900e8 2014-12-23T17:16:38.462+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:38.464+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:38.467+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:38.471+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:38.471+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:38.471+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:38.471+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a37200df4929841900e8 2014-12-23T17:16:38.526+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:38.526+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a37200df4929841900e8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T17:16:38.831+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:38-5499a3769a8f659f34bfb53c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354998831), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:38.904+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:38.904+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:38.904+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2405 2014-12-23T17:16:38.904+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:38.904+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:39.207+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:39.207+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:39-5499a3779a8f659f34bfb53d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354999207), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 433, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:39.298+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37200df4929841900e8') } ntoreturn:1 keyUpdates:0 reslen:37 1500ms 2014-12-23T17:16:39.300+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:39.300+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37200df4929841900e8') } 2014-12-23T17:16:39.907+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3779a8f659f34bfb53e 2014-12-23T17:16:39.907+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:39-5499a3779a8f659f34bfb53f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419354999907), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:39.966+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 2|0||5499a37200df4929841900e8, current metadata version is 2|0||5499a37200df4929841900e8 2014-12-23T17:16:39.967+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-14 from shard version 2|0||5499a37200df4929841900e8 to shard version 2|1||5499a37200df4929841900e8 2014-12-23T17:16:39.967+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a37200df4929841900e8, took 0ms 2014-12-23T17:16:39.967+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a37200df4929841900e8 2014-12-23T17:16:39.967+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:39.969+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:39.972+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:39.976+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:39.984+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:40.001+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:40.001+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:40.001+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:40.001+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a37200df4929841900e8 2014-12-23T17:16:40.053+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:40.053+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a37200df4929841900e8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T17:16:40.358+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:40-5499a3789a8f659f34bfb540", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355000358), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:40.409+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:40.409+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:40.409+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2413 2014-12-23T17:16:40.409+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:40.409+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:40.712+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:40.712+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:40-5499a3789a8f659f34bfb541", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355000712), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 408, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:40.769+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37200df4929841900e8') } ntoreturn:1 keyUpdates:0 reslen:37 1468ms 2014-12-23T17:16:40.771+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a37200df4929841900e8') } 2014-12-23T17:16:41.377+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3789a8f659f34bfb542 2014-12-23T17:16:41.377+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 3|0||5499a37200df4929841900e8, current metadata version is 3|0||5499a37200df4929841900e8 2014-12-23T17:16:41.378+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-14 from shard version 3|0||5499a37200df4929841900e8 to shard version 3|1||5499a37200df4929841900e8 2014-12-23T17:16:41.378+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a37200df4929841900e8, took 0ms 2014-12-23T17:16:41.378+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a37200df4929841900e8 2014-12-23T17:16:41.682+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:41-5499a3799a8f659f34bfb543", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355001682), what: "split", ns: "benchdb1.COL-14", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a37200df4929841900e8') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a37200df4929841900e8') } } } 2014-12-23T17:16:42.003+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:42.004+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a37200df4929841900e8') } ntoreturn:1 keyUpdates:0 reslen:37 1232ms 2014-12-23T17:16:44.228+0000 I INDEX [conn2] build index on: benchdb1.COL-15 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-15" } 2014-12-23T17:16:44.228+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:44.229+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:44.523+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 1|2||5499a37900df4929841900ed, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:44.523+0000 I SHARDING [conn2] collection benchdb1.COL-15 was previously unsharded, new metadata loaded with shard version 1|2||5499a37900df4929841900ed 2014-12-23T17:16:44.523+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a37900df4929841900ed, took 0ms 2014-12-23T17:16:44.565+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:44.565+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37900df4929841900ed') } 2014-12-23T17:16:45.173+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a37c9a8f659f34bfb545 2014-12-23T17:16:45.173+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:45-5499a37d9a8f659f34bfb546", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355005173), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:45.235+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 1|2||5499a37900df4929841900ed, current metadata version is 1|2||5499a37900df4929841900ed 2014-12-23T17:16:45.236+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-15 already up to date (shard version : 1|2||5499a37900df4929841900ed, took 0ms) 2014-12-23T17:16:45.236+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a37900df4929841900ed 2014-12-23T17:16:45.236+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:45.239+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:45.241+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:45.246+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:45.246+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:45.246+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:45.246+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a37900df4929841900ed 2014-12-23T17:16:45.299+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:45.299+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a37900df4929841900ed through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T17:16:45.605+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:45-5499a37d9a8f659f34bfb547", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355005605), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:45.665+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:45.665+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:45.665+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2434 2014-12-23T17:16:45.665+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:45.665+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:45.968+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:45.968+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:45-5499a37d9a8f659f34bfb548", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355005968), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 419, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:46.018+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37900df4929841900ed') } ntoreturn:1 keyUpdates:0 reslen:37 1453ms 2014-12-23T17:16:46.020+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:46.020+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37900df4929841900ed') } 2014-12-23T17:16:46.628+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a37e9a8f659f34bfb549 2014-12-23T17:16:46.628+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:46-5499a37e9a8f659f34bfb54a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355006628), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:46.689+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 2|0||5499a37900df4929841900ed, current metadata version is 2|0||5499a37900df4929841900ed 2014-12-23T17:16:46.690+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-15 from shard version 2|0||5499a37900df4929841900ed to shard version 2|1||5499a37900df4929841900ed 2014-12-23T17:16:46.690+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a37900df4929841900ed, took 0ms 2014-12-23T17:16:46.690+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a37900df4929841900ed 2014-12-23T17:16:46.690+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:46.693+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:46.695+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:46.699+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:46.708+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:46.724+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:46.724+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:46.725+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:46.725+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a37900df4929841900ed 2014-12-23T17:16:46.779+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:46.779+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a37900df4929841900ed through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T17:16:47.085+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:47-5499a37f9a8f659f34bfb54b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355007085), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:47.135+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:47.135+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:47.135+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2442 2014-12-23T17:16:47.135+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:47.135+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:47.439+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:47.439+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:47-5499a37f9a8f659f34bfb54c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355007439), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 410, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:47.491+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a37900df4929841900ed') } ntoreturn:1 keyUpdates:0 reslen:37 1471ms 2014-12-23T17:16:47.493+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a37900df4929841900ed') } 2014-12-23T17:16:48.100+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a37f9a8f659f34bfb54d 2014-12-23T17:16:48.100+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 3|0||5499a37900df4929841900ed, current metadata version is 3|0||5499a37900df4929841900ed 2014-12-23T17:16:48.101+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-15 from shard version 3|0||5499a37900df4929841900ed to shard version 3|1||5499a37900df4929841900ed 2014-12-23T17:16:48.101+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a37900df4929841900ed, took 0ms 2014-12-23T17:16:48.101+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a37900df4929841900ed 2014-12-23T17:16:48.405+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:48-5499a3809a8f659f34bfb54e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355008405), what: "split", ns: "benchdb1.COL-15", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a37900df4929841900ed') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a37900df4929841900ed') } } } 2014-12-23T17:16:48.775+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:48.775+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a37900df4929841900ed') } ntoreturn:1 keyUpdates:0 reslen:37 1282ms 2014-12-23T17:16:51.349+0000 I INDEX [conn2] build index on: benchdb1.COL-16 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-16" } 2014-12-23T17:16:51.349+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:16:51.350+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:16:51.626+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 1|2||5499a38000df4929841900f0, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:16:51.627+0000 I SHARDING [conn2] collection benchdb1.COL-16 was previously unsharded, new metadata loaded with shard version 1|2||5499a38000df4929841900f0 2014-12-23T17:16:51.627+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a38000df4929841900f0, took 0ms 2014-12-23T17:16:51.666+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:51.666+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a38000df4929841900f0') } 2014-12-23T17:16:52.273+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3839a8f659f34bfb550 2014-12-23T17:16:52.273+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:52-5499a3849a8f659f34bfb551", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355012273), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:16:52.337+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 1|2||5499a38000df4929841900f0, current metadata version is 1|2||5499a38000df4929841900f0 2014-12-23T17:16:52.338+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-16 already up to date (shard version : 1|2||5499a38000df4929841900f0, took 0ms) 2014-12-23T17:16:52.338+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a38000df4929841900f0 2014-12-23T17:16:52.339+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:52.342+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:52.344+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:52.349+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:52.349+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:52.349+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:52.349+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a38000df4929841900f0 2014-12-23T17:16:52.414+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:52.415+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a38000df4929841900f0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T17:16:52.719+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:52-5499a3849a8f659f34bfb552", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355012719), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:52.778+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:52.778+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:52.778+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2461 2014-12-23T17:16:52.778+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:16:52.778+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:53.031+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:53.031+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:53-5499a3859a8f659f34bfb553", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355013031), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 429, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:16:53.094+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a38000df4929841900f0') } ntoreturn:1 keyUpdates:0 reslen:37 1428ms 2014-12-23T17:16:53.096+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:16:53.096+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a38000df4929841900f0') } 2014-12-23T17:16:53.507+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3859a8f659f34bfb554 2014-12-23T17:16:53.507+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:53-5499a3859a8f659f34bfb555", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355013507), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:16:53.562+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 2|0||5499a38000df4929841900f0, current metadata version is 2|0||5499a38000df4929841900f0 2014-12-23T17:16:53.563+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-16 from shard version 2|0||5499a38000df4929841900f0 to shard version 2|1||5499a38000df4929841900f0 2014-12-23T17:16:53.563+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a38000df4929841900f0, took 0ms 2014-12-23T17:16:53.563+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a38000df4929841900f0 2014-12-23T17:16:53.563+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:16:53.565+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:53.568+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:53.572+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:53.581+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:53.597+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:16:53.597+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:16:53.597+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:16:53.597+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a38000df4929841900f0 2014-12-23T17:16:53.655+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:16:53.655+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a38000df4929841900f0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T17:16:53.712+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a38000df4929841900f0') } 2014-12-23T17:16:53.712+0000 W SHARDING [conn1] migration already in progress 2014-12-23T17:16:53.712+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:53-5499a3859a8f659f34bfb556", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355013712), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:16:53.892+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:53-5499a3859a8f659f34bfb557", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355013892), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:16:53.937+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:53.937+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:16:53.937+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2470 2014-12-23T17:16:53.937+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:16:53.937+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:16:54.142+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:54.142+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:54-5499a3869a8f659f34bfb558", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355014142), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 466, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 340, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:16:54.208+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a38000df4929841900f0') } ntoreturn:1 keyUpdates:0 reslen:37 1111ms 2014-12-23T17:16:54.209+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a38000df4929841900f0') } 2014-12-23T17:16:54.815+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a3869a8f659f34bfb559 2014-12-23T17:16:54.815+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 3|0||5499a38000df4929841900f0, current metadata version is 3|0||5499a38000df4929841900f0 2014-12-23T17:16:54.816+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-16 from shard version 3|0||5499a38000df4929841900f0 to shard version 3|1||5499a38000df4929841900f0 2014-12-23T17:16:54.816+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a38000df4929841900f0, took 0ms 2014-12-23T17:16:54.816+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a38000df4929841900f0 2014-12-23T17:16:55.121+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:16:55-5499a3879a8f659f34bfb55a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355015121), what: "split", ns: "benchdb1.COL-16", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a38000df4929841900f0') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a38000df4929841900f0') } } } 2014-12-23T17:16:55.451+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:16:55.451+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a38000df4929841900f0') } ntoreturn:1 keyUpdates:0 reslen:37 1241ms 2014-12-23T17:16:55.551+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:16:55 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:17:26.158+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:17:25 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:17:56.764+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:17:56 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:18:27.371+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:18:26 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:18:57.810+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:18:57 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:19:28.416+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:19:27 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:19:59.023+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:19:58 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:20:29.629+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:20:29 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:21:00.235+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:20:59 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:21:30.841+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:21:30 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:22:01.447+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:22:00 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:22:31.979+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:22:31 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:23:02.586+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:23:01 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:23:33.192+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:23:32 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:24:03.799+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:24:03 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:24:34.357+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:24:33 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:24:59.029+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-0 2014-12-23T17:24:59.095+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a34800df4929841900ce, current metadata version is 3|3||5499a34800df4929841900ce 2014-12-23T17:24:59.095+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-0, previous version was 0|0||5499a34800df4929841900ce, this is a drop 2014-12-23T17:24:59.095+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-0 at shard version 3|3||5499a34800df4929841900ce, took 0ms 2014-12-23T17:25:00.152+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-1 2014-12-23T17:25:00.213+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a34f00df4929841900d3, current metadata version is 3|3||5499a34f00df4929841900d3 2014-12-23T17:25:00.214+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-1, previous version was 0|0||5499a34f00df4929841900d3, this is a drop 2014-12-23T17:25:00.214+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-1 at shard version 3|3||5499a34f00df4929841900d3, took 0ms 2014-12-23T17:25:01.303+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-10 2014-12-23T17:25:01.374+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a35600df4929841900d8, current metadata version is 3|3||5499a35600df4929841900d8 2014-12-23T17:25:01.374+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-10, previous version was 0|0||5499a35600df4929841900d8, this is a drop 2014-12-23T17:25:01.374+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-10 at shard version 3|3||5499a35600df4929841900d8, took 0ms 2014-12-23T17:25:02.440+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-11 2014-12-23T17:25:02.510+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a35d00df4929841900db, current metadata version is 3|3||5499a35d00df4929841900db 2014-12-23T17:25:02.511+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-11, previous version was 0|0||5499a35d00df4929841900db, this is a drop 2014-12-23T17:25:02.511+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-11 at shard version 3|3||5499a35d00df4929841900db, took 0ms 2014-12-23T17:25:03.573+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-12 2014-12-23T17:25:03.641+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a36400df4929841900e0, current metadata version is 3|3||5499a36400df4929841900e0 2014-12-23T17:25:03.642+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-12, previous version was 0|0||5499a36400df4929841900e0, this is a drop 2014-12-23T17:25:03.642+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-12 at shard version 3|3||5499a36400df4929841900e0, took 0ms 2014-12-23T17:25:04.712+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-13 2014-12-23T17:25:04.799+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a36b00df4929841900e4, current metadata version is 3|3||5499a36b00df4929841900e4 2014-12-23T17:25:04.800+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-13, previous version was 0|0||5499a36b00df4929841900e4, this is a drop 2014-12-23T17:25:04.800+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-13 at shard version 3|3||5499a36b00df4929841900e4, took 0ms 2014-12-23T17:25:04.852+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:25:04 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:25:05.866+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-14 2014-12-23T17:25:05.930+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a37200df4929841900e8, current metadata version is 3|3||5499a37200df4929841900e8 2014-12-23T17:25:05.931+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-14, previous version was 0|0||5499a37200df4929841900e8, this is a drop 2014-12-23T17:25:05.931+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-14 at shard version 3|3||5499a37200df4929841900e8, took 0ms 2014-12-23T17:25:06.895+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-15 2014-12-23T17:25:07.012+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a37900df4929841900ed, current metadata version is 3|3||5499a37900df4929841900ed 2014-12-23T17:25:07.013+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-15, previous version was 0|0||5499a37900df4929841900ed, this is a drop 2014-12-23T17:25:07.013+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-15 at shard version 3|3||5499a37900df4929841900ed, took 0ms 2014-12-23T17:25:08.031+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-16 2014-12-23T17:25:08.086+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a38000df4929841900f0, current metadata version is 3|3||5499a38000df4929841900f0 2014-12-23T17:25:08.087+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-16, previous version was 0|0||5499a38000df4929841900f0, this is a drop 2014-12-23T17:25:08.087+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-16 at shard version 3|3||5499a38000df4929841900f0, took 0ms 2014-12-23T17:25:08.483+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:25:08.483+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:25:08.640+0000 I INDEX [conn2] build index on: benchdb1.COL-0 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-0" } 2014-12-23T17:25:08.640+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:08.641+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:08.926+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 1|2||5499a57100df49298419016a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:08.927+0000 I SHARDING [conn2] collection benchdb1.COL-0 was previously unsharded, new metadata loaded with shard version 1|2||5499a57100df49298419016a 2014-12-23T17:25:08.927+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a57100df49298419016a, took 0ms 2014-12-23T17:25:08.969+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:08.969+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57100df49298419016a') } 2014-12-23T17:25:09.578+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5749a8f659f34bfb55c 2014-12-23T17:25:09.578+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:09-5499a5759a8f659f34bfb55d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355509578), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:09.638+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 1|2||5499a57100df49298419016a, current metadata version is 1|2||5499a57100df49298419016a 2014-12-23T17:25:09.639+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-0 already up to date (shard version : 1|2||5499a57100df49298419016a, took 0ms) 2014-12-23T17:25:09.639+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a57100df49298419016a 2014-12-23T17:25:09.639+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:09.642+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:09.644+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:09.649+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:09.649+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:09.649+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:09.649+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a57100df49298419016a 2014-12-23T17:25:09.718+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:09.718+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a57100df49298419016a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:25:10.007+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:10-5499a5769a8f659f34bfb55e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355510007), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:10.059+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:10.059+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:10.059+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2610 2014-12-23T17:25:10.059+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:10.059+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:10.362+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:10.362+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:10-5499a5769a8f659f34bfb55f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355510362), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 410, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:10.419+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57100df49298419016a') } ntoreturn:1 keyUpdates:0 reslen:37 1449ms 2014-12-23T17:25:10.420+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:10.421+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57100df49298419016a') } 2014-12-23T17:25:11.027+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5769a8f659f34bfb560 2014-12-23T17:25:11.027+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:11-5499a5779a8f659f34bfb561", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355511027), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:25:11.092+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 2|0||5499a57100df49298419016a, current metadata version is 2|0||5499a57100df49298419016a 2014-12-23T17:25:11.093+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 2|0||5499a57100df49298419016a to shard version 2|1||5499a57100df49298419016a 2014-12-23T17:25:11.093+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a57100df49298419016a, took 0ms 2014-12-23T17:25:11.093+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a57100df49298419016a 2014-12-23T17:25:11.093+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:11.095+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:11.098+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:11.102+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:11.111+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:11.127+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:11.127+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:11.127+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:11.128+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a57100df49298419016a 2014-12-23T17:25:11.181+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:11.181+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a57100df49298419016a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:25:11.487+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:11-5499a5779a8f659f34bfb562", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355511487), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:11.553+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:11.553+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:11.553+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2618 2014-12-23T17:25:11.553+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:25:11.553+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:11.857+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:11.857+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:11-5499a5779a8f659f34bfb563", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355511857), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 425, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:25:11.915+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57100df49298419016a') } ntoreturn:1 keyUpdates:0 reslen:37 1494ms 2014-12-23T17:25:11.917+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a57100df49298419016a') } 2014-12-23T17:25:12.526+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5779a8f659f34bfb564 2014-12-23T17:25:12.526+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 3|0||5499a57100df49298419016a, current metadata version is 3|0||5499a57100df49298419016a 2014-12-23T17:25:12.527+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 3|0||5499a57100df49298419016a to shard version 3|1||5499a57100df49298419016a 2014-12-23T17:25:12.527+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a57100df49298419016a, took 0ms 2014-12-23T17:25:12.527+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a57100df49298419016a 2014-12-23T17:25:12.832+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:12-5499a5789a8f659f34bfb565", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355512832), what: "split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a57100df49298419016a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a57100df49298419016a') } } } 2014-12-23T17:25:13.207+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:13.207+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a57100df49298419016a') } ntoreturn:1 keyUpdates:0 reslen:37 1289ms 2014-12-23T17:25:15.800+0000 I INDEX [conn2] build index on: benchdb1.COL-1 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-1" } 2014-12-23T17:25:15.800+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:15.801+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:16.104+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 1|2||5499a57800df49298419016d, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:16.105+0000 I SHARDING [conn2] collection benchdb1.COL-1 was previously unsharded, new metadata loaded with shard version 1|2||5499a57800df49298419016d 2014-12-23T17:25:16.105+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a57800df49298419016d, took 0ms 2014-12-23T17:25:16.144+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:16.144+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57800df49298419016d') } 2014-12-23T17:25:16.752+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a57c9a8f659f34bfb567 2014-12-23T17:25:16.752+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:16-5499a57c9a8f659f34bfb568", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355516752), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:16.814+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||5499a57800df49298419016d, current metadata version is 1|2||5499a57800df49298419016d 2014-12-23T17:25:16.815+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||5499a57800df49298419016d, took 0ms) 2014-12-23T17:25:16.815+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a57800df49298419016d 2014-12-23T17:25:16.815+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:16.818+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:16.820+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:16.824+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:16.824+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:16.825+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:16.825+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a57800df49298419016d 2014-12-23T17:25:16.881+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:16.881+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a57800df49298419016d through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:25:17.186+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:17-5499a57d9a8f659f34bfb569", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355517186), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:17.245+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:17.245+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:17.245+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2637 2014-12-23T17:25:17.245+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:17.245+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:17.548+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:17.548+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:17-5499a57d9a8f659f34bfb56a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355517548), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 420, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:17.606+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57800df49298419016d') } ntoreturn:1 keyUpdates:0 reslen:37 1461ms 2014-12-23T17:25:17.607+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:17.608+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57800df49298419016d') } 2014-12-23T17:25:18.215+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a57d9a8f659f34bfb56b 2014-12-23T17:25:18.215+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:18-5499a57e9a8f659f34bfb56c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355518215), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:25:18.273+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 2|0||5499a57800df49298419016d, current metadata version is 2|0||5499a57800df49298419016d 2014-12-23T17:25:18.274+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 2|0||5499a57800df49298419016d to shard version 2|1||5499a57800df49298419016d 2014-12-23T17:25:18.274+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a57800df49298419016d, took 0ms 2014-12-23T17:25:18.274+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a57800df49298419016d 2014-12-23T17:25:18.274+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:18.277+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:18.279+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:18.284+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:18.292+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:18.309+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:18.309+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:18.309+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:18.309+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a57800df49298419016d 2014-12-23T17:25:18.360+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:18.360+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a57800df49298419016d through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:25:18.636+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:18-5499a57e9a8f659f34bfb56d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355518636), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:18.706+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:18.706+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:18.706+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2645 2014-12-23T17:25:18.706+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:25:18.707+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:18.959+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:18.959+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:18-5499a57e9a8f659f34bfb56e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355518959), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 397, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:25:19.017+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57800df49298419016d') } ntoreturn:1 keyUpdates:0 reslen:37 1410ms 2014-12-23T17:25:19.019+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a57800df49298419016d') } 2014-12-23T17:25:19.478+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a57f9a8f659f34bfb56f 2014-12-23T17:25:19.478+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 3|0||5499a57800df49298419016d, current metadata version is 3|0||5499a57800df49298419016d 2014-12-23T17:25:19.478+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 3|0||5499a57800df49298419016d to shard version 3|1||5499a57800df49298419016d 2014-12-23T17:25:19.479+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a57800df49298419016d, took 0ms 2014-12-23T17:25:19.479+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a57800df49298419016d 2014-12-23T17:25:19.783+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:19-5499a57f9a8f659f34bfb570", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355519783), what: "split", ns: "benchdb1.COL-1", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a57800df49298419016d') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a57800df49298419016d') } } } 2014-12-23T17:25:20.156+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:20.156+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a57800df49298419016d') } ntoreturn:1 keyUpdates:0 reslen:37 1136ms 2014-12-23T17:25:22.708+0000 I INDEX [conn2] build index on: benchdb1.COL-2 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-2" } 2014-12-23T17:25:22.708+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:22.708+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:23.029+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-2 with requested shard version 1|2||5499a57f00df492984190172, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:23.030+0000 I SHARDING [conn2] collection benchdb1.COL-2 was previously unsharded, new metadata loaded with shard version 1|2||5499a57f00df492984190172 2014-12-23T17:25:23.030+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a57f00df492984190172, took 0ms 2014-12-23T17:25:23.067+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:23.068+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57f00df492984190172') } 2014-12-23T17:25:23.674+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5839a8f659f34bfb572 2014-12-23T17:25:23.674+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:23-5499a5839a8f659f34bfb573", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355523674), what: "moveChunk.start", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:23.734+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 1|2||5499a57f00df492984190172, current metadata version is 1|2||5499a57f00df492984190172 2014-12-23T17:25:23.735+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-2 already up to date (shard version : 1|2||5499a57f00df492984190172, took 0ms) 2014-12-23T17:25:23.735+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a57f00df492984190172 2014-12-23T17:25:23.736+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:23.738+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:23.741+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:23.745+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:23.745+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:23.745+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:23.745+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a57f00df492984190172 2014-12-23T17:25:23.796+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:23.797+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a57f00df492984190172 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-2' 2014-12-23T17:25:24.101+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:24-5499a5849a8f659f34bfb574", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355524101), what: "moveChunk.commit", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:24.154+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:24.154+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:24.154+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-2 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2666 2014-12-23T17:25:24.154+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-2 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:24.154+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:24.457+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:24.457+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:24-5499a5849a8f659f34bfb575", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355524457), what: "moveChunk.from", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 667, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 408, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:24.509+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57f00df492984190172') } ntoreturn:1 keyUpdates:0 reslen:37 1441ms 2014-12-23T17:25:24.511+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:24.511+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57f00df492984190172') } 2014-12-23T17:25:25.118+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5849a8f659f34bfb576 2014-12-23T17:25:25.118+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:25-5499a5859a8f659f34bfb577", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355525118), what: "moveChunk.start", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:25:25.184+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 2|0||5499a57f00df492984190172, current metadata version is 2|0||5499a57f00df492984190172 2014-12-23T17:25:25.185+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-2 from shard version 2|0||5499a57f00df492984190172 to shard version 2|1||5499a57f00df492984190172 2014-12-23T17:25:25.185+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a57f00df492984190172, took 0ms 2014-12-23T17:25:25.185+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a57f00df492984190172 2014-12-23T17:25:25.185+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:25.187+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:25.190+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:25.194+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:25.203+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:25.219+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:25.219+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:25.219+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:25.219+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a57f00df492984190172 2014-12-23T17:25:25.271+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:25.271+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a57f00df492984190172 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-2' 2014-12-23T17:25:25.576+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:25-5499a5859a8f659f34bfb578", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355525576), what: "moveChunk.commit", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:25.643+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:25.643+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:25.643+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-2 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2674 2014-12-23T17:25:25.643+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-2 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:25:25.644+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:25.947+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:25.947+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:25-5499a5859a8f659f34bfb579", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355525947), what: "moveChunk.from", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:25:25.999+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a57f00df492984190172') } ntoreturn:1 keyUpdates:0 reslen:37 1488ms 2014-12-23T17:25:26.001+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-2", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-2-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a57f00df492984190172') } 2014-12-23T17:25:26.608+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5859a8f659f34bfb57a 2014-12-23T17:25:26.608+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 3|0||5499a57f00df492984190172, current metadata version is 3|0||5499a57f00df492984190172 2014-12-23T17:25:26.609+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-2 from shard version 3|0||5499a57f00df492984190172 to shard version 3|1||5499a57f00df492984190172 2014-12-23T17:25:26.609+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a57f00df492984190172, took 0ms 2014-12-23T17:25:26.609+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a57f00df492984190172 2014-12-23T17:25:26.914+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:26-5499a5869a8f659f34bfb57b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355526914), what: "split", ns: "benchdb1.COL-2", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a57f00df492984190172') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a57f00df492984190172') } } } 2014-12-23T17:25:27.283+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:27.283+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-2", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-2-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a57f00df492984190172') } ntoreturn:1 keyUpdates:0 reslen:37 1281ms 2014-12-23T17:25:29.836+0000 I INDEX [conn2] build index on: benchdb1.COL-3 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-3" } 2014-12-23T17:25:29.836+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:29.837+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:30.202+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-3 with requested shard version 1|2||5499a58600df492984190176, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:30.203+0000 I SHARDING [conn2] collection benchdb1.COL-3 was previously unsharded, new metadata loaded with shard version 1|2||5499a58600df492984190176 2014-12-23T17:25:30.203+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a58600df492984190176, took 0ms 2014-12-23T17:25:30.246+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:30.246+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58600df492984190176') } 2014-12-23T17:25:30.724+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a58a9a8f659f34bfb57d 2014-12-23T17:25:30.724+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:30-5499a58a9a8f659f34bfb57e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355530724), what: "moveChunk.start", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:30.793+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 1|2||5499a58600df492984190176, current metadata version is 1|2||5499a58600df492984190176 2014-12-23T17:25:30.794+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-3 already up to date (shard version : 1|2||5499a58600df492984190176, took 0ms) 2014-12-23T17:25:30.794+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a58600df492984190176 2014-12-23T17:25:30.795+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:30.797+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:30.800+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:30.804+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:30.804+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:30.805+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:30.805+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a58600df492984190176 2014-12-23T17:25:30.858+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:30.858+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a58600df492984190176 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-3' 2014-12-23T17:25:31.163+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:31-5499a58b9a8f659f34bfb57f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355531163), what: "moveChunk.commit", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:31.211+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:31.211+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:31.211+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-3 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2695 2014-12-23T17:25:31.211+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-3 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:31.211+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:31.514+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:31.514+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:31-5499a58b9a8f659f34bfb580", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355531514), what: "moveChunk.from", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 548, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 406, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:31.577+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58600df492984190176') } ntoreturn:1 keyUpdates:0 reslen:37 1331ms 2014-12-23T17:25:31.579+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:31.579+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58600df492984190176') } 2014-12-23T17:25:32.187+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a58b9a8f659f34bfb581 2014-12-23T17:25:32.187+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:32-5499a58c9a8f659f34bfb582", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355532187), what: "moveChunk.start", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:25:32.248+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 2|0||5499a58600df492984190176, current metadata version is 2|0||5499a58600df492984190176 2014-12-23T17:25:32.249+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-3 from shard version 2|0||5499a58600df492984190176 to shard version 2|1||5499a58600df492984190176 2014-12-23T17:25:32.249+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a58600df492984190176, took 0ms 2014-12-23T17:25:32.249+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a58600df492984190176 2014-12-23T17:25:32.250+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:32.252+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:32.255+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:32.259+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:32.267+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:32.284+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:32.284+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:32.284+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:32.284+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a58600df492984190176 2014-12-23T17:25:32.340+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:32.341+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a58600df492984190176 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-3' 2014-12-23T17:25:32.646+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:32-5499a58c9a8f659f34bfb583", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355532646), what: "moveChunk.commit", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:32.709+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:32.709+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:32.709+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-3 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2703 2014-12-23T17:25:32.709+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-3 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:25:32.709+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:33.012+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:33.012+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:33-5499a58d9a8f659f34bfb584", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355533012), what: "moveChunk.from", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 425, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:25:33.071+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58600df492984190176') } ntoreturn:1 keyUpdates:0 reslen:37 1492ms 2014-12-23T17:25:33.073+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-3", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-3-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a58600df492984190176') } 2014-12-23T17:25:33.681+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a58d9a8f659f34bfb585 2014-12-23T17:25:33.681+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 3|0||5499a58600df492984190176, current metadata version is 3|0||5499a58600df492984190176 2014-12-23T17:25:33.682+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-3 from shard version 3|0||5499a58600df492984190176 to shard version 3|1||5499a58600df492984190176 2014-12-23T17:25:33.682+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a58600df492984190176, took 0ms 2014-12-23T17:25:33.682+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a58600df492984190176 2014-12-23T17:25:33.987+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:33-5499a58d9a8f659f34bfb586", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355533987), what: "split", ns: "benchdb1.COL-3", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a58600df492984190176') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a58600df492984190176') } } } 2014-12-23T17:25:34.357+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:34.357+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-3", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-3-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a58600df492984190176') } ntoreturn:1 keyUpdates:0 reslen:37 1284ms 2014-12-23T17:25:35.406+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:25:34 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:25:36.954+0000 I INDEX [conn2] build index on: benchdb1.COL-4 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-4" } 2014-12-23T17:25:36.954+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:36.955+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:37.295+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-4 with requested shard version 1|2||5499a58d00df49298419017a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:37.295+0000 I SHARDING [conn2] collection benchdb1.COL-4 was previously unsharded, new metadata loaded with shard version 1|2||5499a58d00df49298419017a 2014-12-23T17:25:37.295+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a58d00df49298419017a, took 0ms 2014-12-23T17:25:37.340+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:37.340+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58d00df49298419017a') } 2014-12-23T17:25:37.947+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5919a8f659f34bfb588 2014-12-23T17:25:37.947+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:37-5499a5919a8f659f34bfb589", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355537947), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:38.012+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 1|2||5499a58d00df49298419017a, current metadata version is 1|2||5499a58d00df49298419017a 2014-12-23T17:25:38.013+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-4 already up to date (shard version : 1|2||5499a58d00df49298419017a, took 0ms) 2014-12-23T17:25:38.013+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a58d00df49298419017a 2014-12-23T17:25:38.013+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:38.016+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:38.018+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:38.023+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:38.023+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:38.023+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:38.023+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a58d00df49298419017a 2014-12-23T17:25:38.075+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:38.075+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a58d00df49298419017a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-4' 2014-12-23T17:25:38.379+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:38-5499a5929a8f659f34bfb58a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355538379), what: "moveChunk.commit", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:38.436+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:38.436+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:38.436+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-4 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2722 2014-12-23T17:25:38.436+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-4 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:38.437+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:38.740+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:38.740+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:38-5499a5929a8f659f34bfb58b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355538740), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 413, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:38.792+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58d00df49298419017a') } ntoreturn:1 keyUpdates:0 reslen:37 1452ms 2014-12-23T17:25:38.794+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:38.794+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58d00df49298419017a') } 2014-12-23T17:25:39.401+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5929a8f659f34bfb58c 2014-12-23T17:25:39.401+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:39-5499a5939a8f659f34bfb58d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355539401), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:25:39.456+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 2|0||5499a58d00df49298419017a, current metadata version is 2|0||5499a58d00df49298419017a 2014-12-23T17:25:39.457+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-4 from shard version 2|0||5499a58d00df49298419017a to shard version 2|1||5499a58d00df49298419017a 2014-12-23T17:25:39.457+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a58d00df49298419017a, took 0ms 2014-12-23T17:25:39.457+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a58d00df49298419017a 2014-12-23T17:25:39.457+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:39.459+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:39.462+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:39.466+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:39.475+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:39.491+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:39.491+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:39.491+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:39.491+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a58d00df49298419017a 2014-12-23T17:25:39.549+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:39.549+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a58d00df49298419017a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-4' 2014-12-23T17:25:39.854+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:39-5499a5939a8f659f34bfb58e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355539854), what: "moveChunk.commit", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:39.915+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:39.915+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:39.915+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-4 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2730 2014-12-23T17:25:39.915+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-4 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:25:39.915+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:40.218+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:40.218+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:40-5499a5949a8f659f34bfb58f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355540218), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:25:40.273+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a58d00df49298419017a') } ntoreturn:1 keyUpdates:0 reslen:37 1478ms 2014-12-23T17:25:40.275+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a58d00df49298419017a') } 2014-12-23T17:25:40.754+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5949a8f659f34bfb590 2014-12-23T17:25:40.754+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 3|0||5499a58d00df49298419017a, current metadata version is 3|0||5499a58d00df49298419017a 2014-12-23T17:25:40.755+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-4 from shard version 3|0||5499a58d00df49298419017a to shard version 3|1||5499a58d00df49298419017a 2014-12-23T17:25:40.755+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a58d00df49298419017a, took 0ms 2014-12-23T17:25:40.755+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a58d00df49298419017a 2014-12-23T17:25:41.061+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:41-5499a5959a8f659f34bfb591", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355541061), what: "split", ns: "benchdb1.COL-4", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a58d00df49298419017a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a58d00df49298419017a') } } } 2014-12-23T17:25:41.279+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:41.279+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a58d00df49298419017a') } ntoreturn:1 keyUpdates:0 reslen:37 1004ms 2014-12-23T17:25:43.670+0000 I INDEX [conn2] build index on: benchdb1.COL-5 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-5" } 2014-12-23T17:25:43.670+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:43.671+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:43.960+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-5 with requested shard version 1|2||5499a59400df49298419017f, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:43.961+0000 I SHARDING [conn2] collection benchdb1.COL-5 was previously unsharded, new metadata loaded with shard version 1|2||5499a59400df49298419017f 2014-12-23T17:25:43.961+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a59400df49298419017f, took 0ms 2014-12-23T17:25:44.000+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:44.001+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59400df49298419017f') } 2014-12-23T17:25:44.607+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5979a8f659f34bfb593 2014-12-23T17:25:44.607+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:44-5499a5989a8f659f34bfb594", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355544607), what: "moveChunk.start", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:44.669+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 1|2||5499a59400df49298419017f, current metadata version is 1|2||5499a59400df49298419017f 2014-12-23T17:25:44.670+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-5 already up to date (shard version : 1|2||5499a59400df49298419017f, took 0ms) 2014-12-23T17:25:44.670+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a59400df49298419017f 2014-12-23T17:25:44.670+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:44.673+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:44.675+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:44.679+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:44.679+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:44.680+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:44.680+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a59400df49298419017f 2014-12-23T17:25:44.733+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:44.733+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a59400df49298419017f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-5' 2014-12-23T17:25:45.037+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:45-5499a5999a8f659f34bfb595", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355545037), what: "moveChunk.commit", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:45.087+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:45.087+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:45.088+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-5 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2750 2014-12-23T17:25:45.088+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-5 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:45.088+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:45.391+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:45.391+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:45-5499a5999a8f659f34bfb596", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355545391), what: "moveChunk.from", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 408, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:45.448+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59400df49298419017f') } ntoreturn:1 keyUpdates:0 reslen:37 1447ms 2014-12-23T17:25:45.449+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:45.450+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59400df49298419017f') } 2014-12-23T17:25:46.056+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5999a8f659f34bfb597 2014-12-23T17:25:46.056+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:46-5499a59a9a8f659f34bfb598", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355546056), what: "moveChunk.start", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:25:46.115+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 2|0||5499a59400df49298419017f, current metadata version is 2|0||5499a59400df49298419017f 2014-12-23T17:25:46.116+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-5 from shard version 2|0||5499a59400df49298419017f to shard version 2|1||5499a59400df49298419017f 2014-12-23T17:25:46.116+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a59400df49298419017f, took 0ms 2014-12-23T17:25:46.116+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a59400df49298419017f 2014-12-23T17:25:46.116+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:46.119+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:46.121+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:46.125+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:46.134+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:46.150+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:46.150+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:46.150+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:46.150+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a59400df49298419017f 2014-12-23T17:25:46.199+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:46.199+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a59400df49298419017f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-5' 2014-12-23T17:25:46.504+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:46-5499a59a9a8f659f34bfb599", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355546504), what: "moveChunk.commit", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:46.563+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:46.563+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:46.563+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-5 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2758 2014-12-23T17:25:46.563+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-5 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:25:46.563+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:46.866+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:46.866+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:46-5499a59a9a8f659f34bfb59a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355546866), what: "moveChunk.from", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 412, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:25:46.922+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59400df49298419017f') } ntoreturn:1 keyUpdates:0 reslen:37 1472ms 2014-12-23T17:25:46.924+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-5", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-5-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a59400df49298419017f') } 2014-12-23T17:25:47.530+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a59a9a8f659f34bfb59b 2014-12-23T17:25:47.530+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 3|0||5499a59400df49298419017f, current metadata version is 3|0||5499a59400df49298419017f 2014-12-23T17:25:47.531+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-5 from shard version 3|0||5499a59400df49298419017f to shard version 3|1||5499a59400df49298419017f 2014-12-23T17:25:47.531+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a59400df49298419017f, took 0ms 2014-12-23T17:25:47.531+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a59400df49298419017f 2014-12-23T17:25:47.836+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:47-5499a59b9a8f659f34bfb59c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355547836), what: "split", ns: "benchdb1.COL-5", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a59400df49298419017f') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a59400df49298419017f') } } } 2014-12-23T17:25:48.205+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:48.205+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-5", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-5-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a59400df49298419017f') } ntoreturn:1 keyUpdates:0 reslen:37 1281ms 2014-12-23T17:25:50.818+0000 I INDEX [conn2] build index on: benchdb1.COL-6 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-6" } 2014-12-23T17:25:50.818+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:50.819+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:51.109+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-6 with requested shard version 1|2||5499a59b00df492984190182, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:51.110+0000 I SHARDING [conn2] collection benchdb1.COL-6 was previously unsharded, new metadata loaded with shard version 1|2||5499a59b00df492984190182 2014-12-23T17:25:51.110+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a59b00df492984190182, took 0ms 2014-12-23T17:25:51.151+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:51.152+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59b00df492984190182') } 2014-12-23T17:25:51.697+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a59f9a8f659f34bfb59e 2014-12-23T17:25:51.697+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:51-5499a59f9a8f659f34bfb59f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355551697), what: "moveChunk.start", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:51.767+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 1|2||5499a59b00df492984190182, current metadata version is 1|2||5499a59b00df492984190182 2014-12-23T17:25:51.767+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-6 already up to date (shard version : 1|2||5499a59b00df492984190182, took 0ms) 2014-12-23T17:25:51.767+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a59b00df492984190182 2014-12-23T17:25:51.768+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:51.771+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:51.773+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:51.777+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:51.777+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:51.778+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:51.778+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a59b00df492984190182 2014-12-23T17:25:51.839+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:51.839+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a59b00df492984190182 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-6' 2014-12-23T17:25:52.143+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:52-5499a5a09a8f659f34bfb5a0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355552143), what: "moveChunk.commit", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:52.198+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:52.198+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:52.198+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-6 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2777 2014-12-23T17:25:52.198+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-6 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:52.198+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:52.368+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:52.368+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:52-5499a5a09a8f659f34bfb5a1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355552368), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 615, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 420, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:52.422+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59b00df492984190182') } ntoreturn:1 keyUpdates:0 reslen:37 1271ms 2014-12-23T17:25:52.424+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:52.424+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59b00df492984190182') } 2014-12-23T17:25:52.454+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a59b00df492984190182') } 2014-12-23T17:25:52.756+0000 I SHARDING [conn4] could not acquire lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' (another update won) 2014-12-23T17:25:52.756+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' was not acquired. 2014-12-23T17:25:52.756+0000 W SHARDING [conn4] could not acquire collection lock for benchdb1.COL-6 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-6 is taken. 2014-12-23T17:25:52.756+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:52-5499a5a09a8f659f34bfb5a4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355552756), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-6 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey..." } } 2014-12-23T17:25:52.809+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a59b00df492984190182') } ntoreturn:1 keyUpdates:0 reslen:245 355ms 2014-12-23T17:25:52.881+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5a09a8f659f34bfb5a2 2014-12-23T17:25:52.881+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:52-5499a5a09a8f659f34bfb5a5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355552881), what: "moveChunk.start", ns: "benchdb1.COL-6", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:25:52.946+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 2|0||5499a59b00df492984190182, current metadata version is 2|0||5499a59b00df492984190182 2014-12-23T17:25:52.947+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-6 from shard version 2|0||5499a59b00df492984190182 to shard version 2|1||5499a59b00df492984190182 2014-12-23T17:25:52.947+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a59b00df492984190182, took 0ms 2014-12-23T17:25:52.947+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a59b00df492984190182 2014-12-23T17:25:52.947+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:52.949+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:52.952+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:52.956+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:52.965+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:52.981+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:52.981+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:52.981+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:52.981+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a59b00df492984190182 2014-12-23T17:25:53.050+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:53.050+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a59b00df492984190182 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-6' 2014-12-23T17:25:53.353+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:53-5499a5a19a8f659f34bfb5a6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355553353), what: "moveChunk.commit", ns: "benchdb1.COL-6", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:53.422+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:53.422+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:53.422+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-6 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2787 2014-12-23T17:25:53.422+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-6 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:25:53.422+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:53.725+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:53.725+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:53-5499a5a19a8f659f34bfb5a7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355553725), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 522, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 441, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:25:53.783+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a59b00df492984190182') } ntoreturn:1 keyUpdates:0 reslen:37 1358ms 2014-12-23T17:25:53.784+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a59b00df492984190182') } 2014-12-23T17:25:54.391+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5a19a8f659f34bfb5a8 2014-12-23T17:25:54.391+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 3|0||5499a59b00df492984190182, current metadata version is 3|0||5499a59b00df492984190182 2014-12-23T17:25:54.392+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-6 from shard version 3|0||5499a59b00df492984190182 to shard version 3|1||5499a59b00df492984190182 2014-12-23T17:25:54.392+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a59b00df492984190182, took 0ms 2014-12-23T17:25:54.392+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a59b00df492984190182 2014-12-23T17:25:54.696+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:54-5499a5a29a8f659f34bfb5a9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355554696), what: "split", ns: "benchdb1.COL-6", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a59b00df492984190182') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a59b00df492984190182') } } } 2014-12-23T17:25:55.065+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:55.065+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a59b00df492984190182') } ntoreturn:1 keyUpdates:0 reslen:37 1280ms 2014-12-23T17:25:57.666+0000 I INDEX [conn2] build index on: benchdb1.COL-7 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-7" } 2014-12-23T17:25:57.666+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:25:57.666+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:25:57.955+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-7 with requested shard version 1|2||5499a5a200df492984190187, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:25:57.955+0000 I SHARDING [conn2] collection benchdb1.COL-7 was previously unsharded, new metadata loaded with shard version 1|2||5499a5a200df492984190187 2014-12-23T17:25:57.955+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5a200df492984190187, took 0ms 2014-12-23T17:25:57.994+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:57.995+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a200df492984190187') } 2014-12-23T17:25:58.601+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5a59a8f659f34bfb5ab 2014-12-23T17:25:58.601+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:58-5499a5a69a8f659f34bfb5ac", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355558601), what: "moveChunk.start", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:25:58.657+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 1|2||5499a5a200df492984190187, current metadata version is 1|2||5499a5a200df492984190187 2014-12-23T17:25:58.658+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-7 already up to date (shard version : 1|2||5499a5a200df492984190187, took 0ms) 2014-12-23T17:25:58.658+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5a200df492984190187 2014-12-23T17:25:58.659+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:25:58.661+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:58.664+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:58.668+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:25:58.668+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:25:58.668+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:25:58.668+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5a200df492984190187 2014-12-23T17:25:58.722+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:25:58.722+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5a200df492984190187 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-7' 2014-12-23T17:25:59.026+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:59-5499a5a79a8f659f34bfb5ad", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355559026), what: "moveChunk.commit", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:25:59.077+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:59.077+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:25:59.077+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-7 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2807 2014-12-23T17:25:59.077+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-7 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:25:59.077+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:25:59.380+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:25:59.380+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:59-5499a5a79a8f659f34bfb5ae", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355559380), what: "moveChunk.from", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 409, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:25:59.439+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a200df492984190187') } ntoreturn:1 keyUpdates:0 reslen:37 1444ms 2014-12-23T17:25:59.440+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:25:59.441+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a200df492984190187') } 2014-12-23T17:25:59.951+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5a79a8f659f34bfb5af 2014-12-23T17:25:59.951+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:25:59-5499a5a79a8f659f34bfb5b0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355559951), what: "moveChunk.start", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:00.007+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 2|0||5499a5a200df492984190187, current metadata version is 2|0||5499a5a200df492984190187 2014-12-23T17:26:00.008+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-7 from shard version 2|0||5499a5a200df492984190187 to shard version 2|1||5499a5a200df492984190187 2014-12-23T17:26:00.008+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5a200df492984190187, took 0ms 2014-12-23T17:26:00.008+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5a200df492984190187 2014-12-23T17:26:00.008+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:00.011+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:00.013+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:00.018+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:00.026+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:00.042+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:00.042+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:00.043+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:00.043+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5a200df492984190187 2014-12-23T17:26:00.095+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:00.096+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5a200df492984190187 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-7' 2014-12-23T17:26:00.400+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:00-5499a5a89a8f659f34bfb5b1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355560400), what: "moveChunk.commit", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:00.461+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:00.461+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:00.461+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-7 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2815 2014-12-23T17:26:00.461+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-7 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:00.461+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:00.724+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:00.724+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:00-5499a5a89a8f659f34bfb5b2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355560724), what: "moveChunk.from", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 567, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 418, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:00.784+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a200df492984190187') } ntoreturn:1 keyUpdates:0 reslen:37 1343ms 2014-12-23T17:26:00.785+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-7-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5a200df492984190187') } 2014-12-23T17:26:01.392+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5a89a8f659f34bfb5b3 2014-12-23T17:26:01.392+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 3|0||5499a5a200df492984190187, current metadata version is 3|0||5499a5a200df492984190187 2014-12-23T17:26:01.393+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-7 from shard version 3|0||5499a5a200df492984190187 to shard version 3|1||5499a5a200df492984190187 2014-12-23T17:26:01.393+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5a200df492984190187, took 0ms 2014-12-23T17:26:01.393+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5a200df492984190187 2014-12-23T17:26:01.697+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:01-5499a5a99a8f659f34bfb5b4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355561697), what: "split", ns: "benchdb1.COL-7", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5a200df492984190187') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5a200df492984190187') } } } 2014-12-23T17:26:02.078+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:02.078+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-7-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5a200df492984190187') } ntoreturn:1 keyUpdates:0 reslen:37 1292ms 2014-12-23T17:26:04.234+0000 I INDEX [conn2] build index on: benchdb1.COL-8 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-8" } 2014-12-23T17:26:04.234+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:04.235+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:04.571+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-8 with requested shard version 1|2||5499a5a900df49298419018c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:04.572+0000 I SHARDING [conn2] collection benchdb1.COL-8 was previously unsharded, new metadata loaded with shard version 1|2||5499a5a900df49298419018c 2014-12-23T17:26:04.572+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5a900df49298419018c, took 0ms 2014-12-23T17:26:04.614+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:04.614+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a900df49298419018c') } 2014-12-23T17:26:05.222+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5ac9a8f659f34bfb5b6 2014-12-23T17:26:05.222+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:05-5499a5ad9a8f659f34bfb5b7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355565222), what: "moveChunk.start", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:05.285+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 1|2||5499a5a900df49298419018c, current metadata version is 1|2||5499a5a900df49298419018c 2014-12-23T17:26:05.286+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-8 already up to date (shard version : 1|2||5499a5a900df49298419018c, took 0ms) 2014-12-23T17:26:05.286+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5a900df49298419018c 2014-12-23T17:26:05.287+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:05.289+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:05.292+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:05.296+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:05.296+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:05.297+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:05.297+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5a900df49298419018c 2014-12-23T17:26:05.353+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:05.353+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5a900df49298419018c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-8' 2014-12-23T17:26:05.659+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:05-5499a5ad9a8f659f34bfb5b8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355565659), what: "moveChunk.commit", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:05.719+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:05.719+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:05.719+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-8 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2836 2014-12-23T17:26:05.719+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-8 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:05.719+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:05.871+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:26:05 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:26:05.971+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:05.972+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:05-5499a5ad9a8f659f34bfb5b9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355565972), what: "moveChunk.from", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 422, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:06.030+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a900df49298419018c') } ntoreturn:1 keyUpdates:0 reslen:37 1416ms 2014-12-23T17:26:06.032+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:06.032+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a900df49298419018c') } 2014-12-23T17:26:06.639+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5ae9a8f659f34bfb5ba 2014-12-23T17:26:06.639+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:06-5499a5ae9a8f659f34bfb5bb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355566639), what: "moveChunk.start", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:06.701+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 2|0||5499a5a900df49298419018c, current metadata version is 2|0||5499a5a900df49298419018c 2014-12-23T17:26:06.702+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-8 from shard version 2|0||5499a5a900df49298419018c to shard version 2|1||5499a5a900df49298419018c 2014-12-23T17:26:06.702+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5a900df49298419018c, took 0ms 2014-12-23T17:26:06.702+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5a900df49298419018c 2014-12-23T17:26:06.702+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:06.705+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:06.707+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:06.711+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:06.720+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:06.736+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:06.736+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:06.737+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:06.737+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5a900df49298419018c 2014-12-23T17:26:06.797+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:06.797+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5a900df49298419018c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-8' 2014-12-23T17:26:07.102+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:07-5499a5af9a8f659f34bfb5bc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355567102), what: "moveChunk.commit", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:07.164+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:07.164+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:07.164+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-8 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2844 2014-12-23T17:26:07.164+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-8 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:07.164+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:07.467+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:07.467+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:07-5499a5af9a8f659f34bfb5bd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355567467), what: "moveChunk.from", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 427, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:07.524+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5a900df49298419018c') } ntoreturn:1 keyUpdates:0 reslen:37 1492ms 2014-12-23T17:26:07.526+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-8-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5a900df49298419018c') } 2014-12-23T17:26:08.084+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5af9a8f659f34bfb5be 2014-12-23T17:26:08.084+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 3|0||5499a5a900df49298419018c, current metadata version is 3|0||5499a5a900df49298419018c 2014-12-23T17:26:08.085+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-8 from shard version 3|0||5499a5a900df49298419018c to shard version 3|1||5499a5a900df49298419018c 2014-12-23T17:26:08.085+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5a900df49298419018c, took 0ms 2014-12-23T17:26:08.085+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5a900df49298419018c 2014-12-23T17:26:08.390+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:08-5499a5b09a8f659f34bfb5bf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355568390), what: "split", ns: "benchdb1.COL-8", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5a900df49298419018c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5a900df49298419018c') } } } 2014-12-23T17:26:08.744+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:08.744+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-8-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5a900df49298419018c') } ntoreturn:1 keyUpdates:0 reslen:37 1217ms 2014-12-23T17:26:11.254+0000 I INDEX [conn2] build index on: benchdb1.COL-9 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-9" } 2014-12-23T17:26:11.254+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:11.255+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:11.557+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-9 with requested shard version 1|2||5499a5b000df49298419018f, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:11.558+0000 I SHARDING [conn2] collection benchdb1.COL-9 was previously unsharded, new metadata loaded with shard version 1|2||5499a5b000df49298419018f 2014-12-23T17:26:11.558+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5b000df49298419018f, took 0ms 2014-12-23T17:26:11.599+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:11.600+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b000df49298419018f') } 2014-12-23T17:26:12.169+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5b39a8f659f34bfb5c1 2014-12-23T17:26:12.169+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:12-5499a5b49a8f659f34bfb5c2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355572169), what: "moveChunk.start", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:12.246+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 1|2||5499a5b000df49298419018f, current metadata version is 1|2||5499a5b000df49298419018f 2014-12-23T17:26:12.247+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-9 already up to date (shard version : 1|2||5499a5b000df49298419018f, took 1ms) 2014-12-23T17:26:12.247+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5b000df49298419018f 2014-12-23T17:26:12.247+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:12.250+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:12.252+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:12.257+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:12.257+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:12.257+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:12.257+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5b000df49298419018f 2014-12-23T17:26:12.321+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:12.321+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5b000df49298419018f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-9' 2014-12-23T17:26:12.626+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:12-5499a5b49a8f659f34bfb5c3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355572626), what: "moveChunk.commit", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:12.672+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:12.672+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:12.672+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-9 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2863 2014-12-23T17:26:12.672+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-9 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:12.672+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:12.927+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:12.927+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:12-5499a5b49a8f659f34bfb5c4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355572927), what: "moveChunk.from", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 647, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 415, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:12.996+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b000df49298419018f') } ntoreturn:1 keyUpdates:0 reslen:37 1396ms 2014-12-23T17:26:12.998+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:12.998+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b000df49298419018f') } 2014-12-23T17:26:13.552+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5b49a8f659f34bfb5c5 2014-12-23T17:26:13.553+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:13-5499a5b59a8f659f34bfb5c6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355573553), what: "moveChunk.start", ns: "benchdb1.COL-9", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:13.610+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 2|0||5499a5b000df49298419018f, current metadata version is 2|0||5499a5b000df49298419018f 2014-12-23T17:26:13.611+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-9 from shard version 2|0||5499a5b000df49298419018f to shard version 2|1||5499a5b000df49298419018f 2014-12-23T17:26:13.611+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5b000df49298419018f, took 0ms 2014-12-23T17:26:13.611+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5b000df49298419018f 2014-12-23T17:26:13.611+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:13.613+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:13.616+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:13.620+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:13.628+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:13.645+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:13.645+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:13.645+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:13.645+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5b000df49298419018f 2014-12-23T17:26:13.701+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:13.701+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5b000df49298419018f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-9' 2014-12-23T17:26:14.006+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:14-5499a5b69a8f659f34bfb5c7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355574006), what: "moveChunk.commit", ns: "benchdb1.COL-9", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:14.068+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:14.068+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:14.068+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-9 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2871 2014-12-23T17:26:14.068+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-9 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:14.068+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:14.344+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:14.344+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:14-5499a5b69a8f659f34bfb5c8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355574344), what: "moveChunk.from", ns: "benchdb1.COL-9", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 612, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:14.405+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b000df49298419018f') } ntoreturn:1 keyUpdates:0 reslen:37 1407ms 2014-12-23T17:26:14.407+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-9-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5b000df49298419018f') } 2014-12-23T17:26:14.904+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5b69a8f659f34bfb5c9 2014-12-23T17:26:14.904+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 3|0||5499a5b000df49298419018f, current metadata version is 3|0||5499a5b000df49298419018f 2014-12-23T17:26:14.905+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-9 from shard version 3|0||5499a5b000df49298419018f to shard version 3|1||5499a5b000df49298419018f 2014-12-23T17:26:14.905+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5b000df49298419018f, took 0ms 2014-12-23T17:26:14.905+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5b000df49298419018f 2014-12-23T17:26:15.113+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:15-5499a5b79a8f659f34bfb5ca", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355575113), what: "split", ns: "benchdb1.COL-9", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5b000df49298419018f') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5b000df49298419018f') } } } 2014-12-23T17:26:15.411+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:15.411+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-9-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5b000df49298419018f') } ntoreturn:1 keyUpdates:0 reslen:37 1004ms 2014-12-23T17:26:18.018+0000 I INDEX [conn2] build index on: benchdb1.COL-10 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-10" } 2014-12-23T17:26:18.018+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:18.019+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:18.322+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 1|2||5499a5b600df492984190194, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:18.323+0000 I SHARDING [conn2] collection benchdb1.COL-10 was previously unsharded, new metadata loaded with shard version 1|2||5499a5b600df492984190194 2014-12-23T17:26:18.323+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5b600df492984190194, took 0ms 2014-12-23T17:26:18.363+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:18.363+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b600df492984190194') } 2014-12-23T17:26:18.970+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5ba9a8f659f34bfb5cc 2014-12-23T17:26:18.970+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:18-5499a5ba9a8f659f34bfb5cd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355578970), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:19.032+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 1|2||5499a5b600df492984190194, current metadata version is 1|2||5499a5b600df492984190194 2014-12-23T17:26:19.032+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-10 already up to date (shard version : 1|2||5499a5b600df492984190194, took 0ms) 2014-12-23T17:26:19.032+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5b600df492984190194 2014-12-23T17:26:19.033+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:19.035+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:19.038+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:19.042+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:19.042+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:19.043+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:19.043+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5b600df492984190194 2014-12-23T17:26:19.099+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:19.099+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5b600df492984190194 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:26:19.373+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:19-5499a5bb9a8f659f34bfb5ce", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355579373), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:19.422+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:19.422+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:19.422+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2892 2014-12-23T17:26:19.423+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:19.423+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:19.726+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:19.726+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:19-5499a5bb9a8f659f34bfb5cf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355579726), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 380, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:19.782+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b600df492984190194') } ntoreturn:1 keyUpdates:0 reslen:37 1419ms 2014-12-23T17:26:19.784+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:19.784+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b600df492984190194') } 2014-12-23T17:26:20.391+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5bb9a8f659f34bfb5d0 2014-12-23T17:26:20.391+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:20-5499a5bc9a8f659f34bfb5d1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355580391), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:20.458+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 2|0||5499a5b600df492984190194, current metadata version is 2|0||5499a5b600df492984190194 2014-12-23T17:26:20.459+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-10 from shard version 2|0||5499a5b600df492984190194 to shard version 2|1||5499a5b600df492984190194 2014-12-23T17:26:20.459+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5b600df492984190194, took 0ms 2014-12-23T17:26:20.459+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5b600df492984190194 2014-12-23T17:26:20.459+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:20.461+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:20.464+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:20.468+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:20.477+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:20.493+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:20.493+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:20.493+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:20.493+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5b600df492984190194 2014-12-23T17:26:20.551+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:20.551+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5b600df492984190194 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:26:20.856+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:20-5499a5bc9a8f659f34bfb5d2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355580856), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:20.916+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:20.916+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:20.916+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2900 2014-12-23T17:26:20.916+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:20.916+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:21.219+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:21.219+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:21-5499a5bd9a8f659f34bfb5d3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355581219), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 674, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:21.275+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5b600df492984190194') } ntoreturn:1 keyUpdates:0 reslen:37 1490ms 2014-12-23T17:26:21.277+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5b600df492984190194') } 2014-12-23T17:26:21.883+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5bd9a8f659f34bfb5d4 2014-12-23T17:26:21.883+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 3|0||5499a5b600df492984190194, current metadata version is 3|0||5499a5b600df492984190194 2014-12-23T17:26:21.884+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-10 from shard version 3|0||5499a5b600df492984190194 to shard version 3|1||5499a5b600df492984190194 2014-12-23T17:26:21.884+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5b600df492984190194, took 0ms 2014-12-23T17:26:21.884+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5b600df492984190194 2014-12-23T17:26:22.189+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:22-5499a5be9a8f659f34bfb5d5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355582189), what: "split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5b600df492984190194') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5b600df492984190194') } } } 2014-12-23T17:26:22.561+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:22.562+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5b600df492984190194') } ntoreturn:1 keyUpdates:0 reslen:37 1284ms 2014-12-23T17:26:25.150+0000 I INDEX [conn2] build index on: benchdb1.COL-11 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-11" } 2014-12-23T17:26:25.150+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:25.151+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:25.478+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 1|2||5499a5be00df492984190197, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:25.479+0000 I SHARDING [conn2] collection benchdb1.COL-11 was previously unsharded, new metadata loaded with shard version 1|2||5499a5be00df492984190197 2014-12-23T17:26:25.479+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5be00df492984190197, took 0ms 2014-12-23T17:26:25.516+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:25.516+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5be00df492984190197') } 2014-12-23T17:26:26.028+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5c19a8f659f34bfb5d7 2014-12-23T17:26:26.028+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:26-5499a5c29a8f659f34bfb5d8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355586028), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:26.073+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 1|2||5499a5be00df492984190197, current metadata version is 1|2||5499a5be00df492984190197 2014-12-23T17:26:26.073+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-11 already up to date (shard version : 1|2||5499a5be00df492984190197, took 0ms) 2014-12-23T17:26:26.073+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5be00df492984190197 2014-12-23T17:26:26.074+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:26.077+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.079+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.083+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.083+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:26.084+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:26.084+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5be00df492984190197 2014-12-23T17:26:26.136+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:26.136+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5be00df492984190197 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T17:26:26.323+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:26-5499a5c29a8f659f34bfb5d9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355586323), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:26.375+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:26.376+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:26.376+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2920 2014-12-23T17:26:26.376+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:26.376+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:26.467+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5be00df492984190197') } 2014-12-23T17:26:26.567+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:26.567+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:26-5499a5c29a8f659f34bfb5db", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355586567), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 557, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 292, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:26.616+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5be00df492984190197') } ntoreturn:1 keyUpdates:0 reslen:37 1099ms 2014-12-23T17:26:26.617+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:26.617+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5be00df492984190197') } 2014-12-23T17:26:26.618+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-11 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-11 is taken. 2014-12-23T17:26:26.618+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:26-5499a5c29a8f659f34bfb5dc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355586618), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-11 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKe..." } } 2014-12-23T17:26:26.682+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5be00df492984190197') } 2014-12-23T17:26:26.683+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-11 to split chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for splitting chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-11 is taken. 2014-12-23T17:26:26.685+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5be00df492984190197') } 2014-12-23T17:26:26.686+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-11 to split chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for splitting chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-11 is taken. 2014-12-23T17:26:26.693+0000 I INDEX [conn2] build index on: benchdb1.COL-12 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-12" } 2014-12-23T17:26:26.693+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:26.693+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:26.829+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5c29a8f659f34bfb5da 2014-12-23T17:26:26.829+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:26-5499a5c29a8f659f34bfb5de", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355586829), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:26.881+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 2|0||5499a5be00df492984190197, current metadata version is 2|0||5499a5be00df492984190197 2014-12-23T17:26:26.882+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-11 from shard version 2|0||5499a5be00df492984190197 to shard version 2|1||5499a5be00df492984190197 2014-12-23T17:26:26.882+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a5be00df492984190197, took 0ms 2014-12-23T17:26:26.882+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a5be00df492984190197 2014-12-23T17:26:26.882+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:26:26.885+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.887+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.891+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.900+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.916+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:26.916+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:26:26.917+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:26:26.917+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a5be00df492984190197 2014-12-23T17:26:26.996+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:26.996+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a5be00df492984190197 through { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } for collection 'benchdb1.COL-11' 2014-12-23T17:26:27.051+0000 I SHARDING [conn2] waiting till out of critical section 2014-12-23T17:26:27.254+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:27-5499a5c39a8f659f34bfb5df", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355587254), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:27.254+0000 I QUERY [conn2] command admin.$cmd command: setShardVersion { setShardVersion: "benchdb1.COL-12", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", shard: "shard0002", shardHost: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", version: Timestamp 1000|2, versionEpoch: ObjectId('5499a5bf00df49298419019b') } ntoreturn:1 keyUpdates:0 reslen:187 202ms 2014-12-23T17:26:27.255+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 1|2||5499a5bf00df49298419019b, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:27.256+0000 I SHARDING [conn2] collection benchdb1.COL-12 was previously unsharded, new metadata loaded with shard version 1|2||5499a5bf00df49298419019b 2014-12-23T17:26:27.256+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5bf00df49298419019b, took 0ms 2014-12-23T17:26:27.306+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:27.307+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:26:27.307+0000 W SHARDING [conn1] migration already in progress 2014-12-23T17:26:27.307+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:27-5499a5c39a8f659f34bfb5e0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355587307), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, to: "shard0001", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:26:27.335+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:27.335+0000 I SHARDING [conn4] forking for cleanup of chunk data 2014-12-23T17:26:27.335+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:27.335+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-11 from { shardkey: MinKey } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T17:26:27.335+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: MinKey } -> { shardkey: -3074457345618258602 } 2014-12-23T17:26:27.386+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:27.386+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:26:27.587+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:27.587+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:27-5499a5c39a8f659f34bfb5e2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355587587), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 415, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 418, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:27.644+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5be00df492984190197') } ntoreturn:1 keyUpdates:0 reslen:37 1176ms 2014-12-23T17:26:27.897+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5c39a8f659f34bfb5e1 2014-12-23T17:26:27.897+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:27-5499a5c39a8f659f34bfb5e3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355587897), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:27.951+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 1|2||5499a5bf00df49298419019b, current metadata version is 1|2||5499a5bf00df49298419019b 2014-12-23T17:26:27.951+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-12 already up to date (shard version : 1|2||5499a5bf00df49298419019b, took 0ms) 2014-12-23T17:26:27.952+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5bf00df49298419019b 2014-12-23T17:26:27.952+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:27.954+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:27.957+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:27.961+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:27.970+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:27.986+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:27.986+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:27.986+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:27.986+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5bf00df49298419019b 2014-12-23T17:26:28.051+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:28.051+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5bf00df49298419019b through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T17:26:28.355+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:28-5499a5c49a8f659f34bfb5e4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355588355), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:28.420+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:28.420+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:28.420+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2951 2014-12-23T17:26:28.420+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:28.420+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:28.723+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:28.723+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:28-5499a5c49a8f659f34bfb5e5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355588723), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 565, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 433, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:28.779+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } ntoreturn:1 keyUpdates:0 reslen:37 1393ms 2014-12-23T17:26:28.781+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:26:29.312+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5c49a8f659f34bfb5e6 2014-12-23T17:26:29.312+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 2|0||5499a5bf00df49298419019b, current metadata version is 2|0||5499a5bf00df49298419019b 2014-12-23T17:26:29.312+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-12 from shard version 2|0||5499a5bf00df49298419019b to shard version 2|1||5499a5bf00df49298419019b 2014-12-23T17:26:29.313+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5bf00df49298419019b, took 0ms 2014-12-23T17:26:29.313+0000 I SHARDING [conn1] splitChunk accepted at version 2|1||5499a5bf00df49298419019b 2014-12-23T17:26:29.492+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:29-5499a5c59a8f659f34bfb5e7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355589492), what: "split", ns: "benchdb1.COL-12", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('5499a5bf00df49298419019b') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('5499a5bf00df49298419019b') } } } 2014-12-23T17:26:29.692+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:29.692+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5bf00df49298419019b') } ntoreturn:1 keyUpdates:0 reslen:37 911ms 2014-12-23T17:26:29.694+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: 0 } ], shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:26:29.913+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:26:29.914+0000 W SHARDING [conn4] could not acquire collection lock for benchdb1.COL-12 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-12 is taken. 2014-12-23T17:26:29.914+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:29-5499a5c59a8f659f34bfb5e9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355589914), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, to: "shard0001", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-12 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKe..." } } 2014-12-23T17:26:30.005+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5c59a8f659f34bfb5e8 2014-12-23T17:26:30.005+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 2|3||5499a5bf00df49298419019b, current metadata version is 2|3||5499a5bf00df49298419019b 2014-12-23T17:26:30.005+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-12 already up to date (shard version : 2|3||5499a5bf00df49298419019b, took 0ms) 2014-12-23T17:26:30.005+0000 I SHARDING [conn1] splitChunk accepted at version 2|3||5499a5bf00df49298419019b 2014-12-23T17:26:30.270+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:30-5499a5c69a8f659f34bfb5ea", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355590270), what: "split", ns: "benchdb1.COL-12", details: { before: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 } }, left: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, lastmod: Timestamp 2000|4, lastmodEpoch: ObjectId('5499a5bf00df49298419019b') }, right: { min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, lastmod: Timestamp 2000|5, lastmodEpoch: ObjectId('5499a5bf00df49298419019b') } } } 2014-12-23T17:26:30.655+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:30.655+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: 0 } ], shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5bf00df49298419019b') } ntoreturn:1 keyUpdates:0 reslen:37 961ms 2014-12-23T17:26:31.973+0000 I INDEX [conn2] build index on: benchdb1.COL-13 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-13" } 2014-12-23T17:26:31.973+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:31.973+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:32.264+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 1|2||5499a5c400df4929841901a1, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:32.265+0000 I SHARDING [conn2] collection benchdb1.COL-13 was previously unsharded, new metadata loaded with shard version 1|2||5499a5c400df4929841901a1 2014-12-23T17:26:32.265+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5c400df4929841901a1, took 0ms 2014-12-23T17:26:32.308+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:32.308+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5c400df4929841901a1') } 2014-12-23T17:26:32.915+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5c89a8f659f34bfb5ec 2014-12-23T17:26:32.915+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:32-5499a5c89a8f659f34bfb5ed", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355592915), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:32.977+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 1|2||5499a5c400df4929841901a1, current metadata version is 1|2||5499a5c400df4929841901a1 2014-12-23T17:26:32.978+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-13 already up to date (shard version : 1|2||5499a5c400df4929841901a1, took 0ms) 2014-12-23T17:26:32.978+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5c400df4929841901a1 2014-12-23T17:26:32.978+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:32.981+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:32.983+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:32.988+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:32.988+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:32.988+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:32.988+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5c400df4929841901a1 2014-12-23T17:26:33.042+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:33.042+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5c400df4929841901a1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T17:26:33.347+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:33-5499a5c99a8f659f34bfb5ee", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355593347), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:33.408+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:33.408+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:33.408+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 2974 2014-12-23T17:26:33.409+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:33.409+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:33.712+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:33.712+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:33-5499a5c99a8f659f34bfb5ef", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355593712), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 420, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:33.764+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5c400df4929841901a1') } ntoreturn:1 keyUpdates:0 reslen:37 1456ms 2014-12-23T17:26:33.765+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:33.766+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5c400df4929841901a1') } 2014-12-23T17:26:34.373+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5c99a8f659f34bfb5f0 2014-12-23T17:26:34.373+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:34-5499a5ca9a8f659f34bfb5f1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355594373), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:34.438+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 2|0||5499a5c400df4929841901a1, current metadata version is 2|0||5499a5c400df4929841901a1 2014-12-23T17:26:34.439+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-13 from shard version 2|0||5499a5c400df4929841901a1 to shard version 2|1||5499a5c400df4929841901a1 2014-12-23T17:26:34.439+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5c400df4929841901a1, took 0ms 2014-12-23T17:26:34.439+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5c400df4929841901a1 2014-12-23T17:26:34.439+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:34.441+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:34.444+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:34.448+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:34.456+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:34.473+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:34.473+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:34.473+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:34.473+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5c400df4929841901a1 2014-12-23T17:26:34.539+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:34.540+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5c400df4929841901a1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T17:26:34.844+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:34-5499a5ca9a8f659f34bfb5f2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355594844), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:34.903+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:34.903+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:34.903+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 2982 2014-12-23T17:26:34.903+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:34.903+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:35.206+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:35.206+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:35-5499a5cb9a8f659f34bfb5f3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355595206), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 430, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:35.261+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5c400df4929841901a1') } ntoreturn:1 keyUpdates:0 reslen:37 1496ms 2014-12-23T17:26:35.263+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5c400df4929841901a1') } 2014-12-23T17:26:35.870+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5cb9a8f659f34bfb5f4 2014-12-23T17:26:35.870+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 3|0||5499a5c400df4929841901a1, current metadata version is 3|0||5499a5c400df4929841901a1 2014-12-23T17:26:35.871+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-13 from shard version 3|0||5499a5c400df4929841901a1 to shard version 3|1||5499a5c400df4929841901a1 2014-12-23T17:26:35.871+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5c400df4929841901a1, took 0ms 2014-12-23T17:26:35.871+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5c400df4929841901a1 2014-12-23T17:26:36.175+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:36-5499a5cc9a8f659f34bfb5f5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355596175), what: "split", ns: "benchdb1.COL-13", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5c400df4929841901a1') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5c400df4929841901a1') } } } 2014-12-23T17:26:36.402+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:26:35 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:26:36.553+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:36.553+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5c400df4929841901a1') } ntoreturn:1 keyUpdates:0 reslen:37 1289ms 2014-12-23T17:26:39.042+0000 I INDEX [conn2] build index on: benchdb1.COL-14 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-14" } 2014-12-23T17:26:39.042+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:39.043+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:39.319+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 1|2||5499a5cb00df4929841901a4, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:39.320+0000 I SHARDING [conn2] collection benchdb1.COL-14 was previously unsharded, new metadata loaded with shard version 1|2||5499a5cb00df4929841901a4 2014-12-23T17:26:39.320+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5cb00df4929841901a4, took 0ms 2014-12-23T17:26:39.359+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:39.359+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5cb00df4929841901a4') } 2014-12-23T17:26:39.966+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5cf9a8f659f34bfb5f7 2014-12-23T17:26:39.966+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:39-5499a5cf9a8f659f34bfb5f8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355599966), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:40.027+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 1|2||5499a5cb00df4929841901a4, current metadata version is 1|2||5499a5cb00df4929841901a4 2014-12-23T17:26:40.028+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-14 already up to date (shard version : 1|2||5499a5cb00df4929841901a4, took 0ms) 2014-12-23T17:26:40.028+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5cb00df4929841901a4 2014-12-23T17:26:40.028+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:40.031+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:40.033+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:40.037+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:40.037+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:40.038+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:40.038+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5cb00df4929841901a4 2014-12-23T17:26:40.090+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:40.090+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5cb00df4929841901a4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T17:26:40.379+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:40-5499a5d09a8f659f34bfb5f9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355600379), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:40.447+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:40.447+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:40.447+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3000 2014-12-23T17:26:40.447+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:40.447+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:40.655+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:40.655+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:40-5499a5d09a8f659f34bfb5fa", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355600655), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 409, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:40.709+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5cb00df4929841901a4') } ntoreturn:1 keyUpdates:0 reslen:37 1350ms 2014-12-23T17:26:40.710+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:40.711+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5cb00df4929841901a4') } 2014-12-23T17:26:41.003+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5d09a8f659f34bfb5fb 2014-12-23T17:26:41.003+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:41-5499a5d19a8f659f34bfb5fc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355601003), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:41.059+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 2|0||5499a5cb00df4929841901a4, current metadata version is 2|0||5499a5cb00df4929841901a4 2014-12-23T17:26:41.060+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-14 from shard version 2|0||5499a5cb00df4929841901a4 to shard version 2|1||5499a5cb00df4929841901a4 2014-12-23T17:26:41.060+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5cb00df4929841901a4, took 0ms 2014-12-23T17:26:41.060+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5cb00df4929841901a4 2014-12-23T17:26:41.060+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:41.062+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:41.065+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:41.069+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:41.078+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:41.094+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:41.094+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:41.094+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:41.095+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5cb00df4929841901a4 2014-12-23T17:26:41.154+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:41.154+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5cb00df4929841901a4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T17:26:41.185+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:26:41.185+0000 W SHARDING [conn4] migration already in progress 2014-12-23T17:26:41.185+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:41-5499a5d19a8f659f34bfb5fd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355601185), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, to: "shard0001", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:26:41.213+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5cb00df4929841901a4') } 2014-12-23T17:26:41.213+0000 W SHARDING [conn4] migration already in progress 2014-12-23T17:26:41.213+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:41-5499a5d19a8f659f34bfb5fe", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355601213), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:26:41.407+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:41-5499a5d19a8f659f34bfb5ff", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355601407), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:41.456+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:41.456+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:41.456+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3010 2014-12-23T17:26:41.456+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:41.456+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:41.660+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:41.660+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:41-5499a5d19a8f659f34bfb600", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355601660), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 349, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 361, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:41.721+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5cb00df4929841901a4') } ntoreturn:1 keyUpdates:0 reslen:37 1010ms 2014-12-23T17:26:41.722+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5cb00df4929841901a4') } 2014-12-23T17:26:42.329+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5d19a8f659f34bfb601 2014-12-23T17:26:42.329+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 3|0||5499a5cb00df4929841901a4, current metadata version is 3|0||5499a5cb00df4929841901a4 2014-12-23T17:26:42.330+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-14 from shard version 3|0||5499a5cb00df4929841901a4 to shard version 3|1||5499a5cb00df4929841901a4 2014-12-23T17:26:42.330+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5cb00df4929841901a4, took 0ms 2014-12-23T17:26:42.330+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5cb00df4929841901a4 2014-12-23T17:26:42.635+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:42-5499a5d29a8f659f34bfb602", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355602635), what: "split", ns: "benchdb1.COL-14", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5cb00df4929841901a4') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5cb00df4929841901a4') } } } 2014-12-23T17:26:43.013+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:43.013+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5cb00df4929841901a4') } ntoreturn:1 keyUpdates:0 reslen:37 1290ms 2014-12-23T17:26:45.510+0000 I INDEX [conn2] build index on: benchdb1.COL-15 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-15" } 2014-12-23T17:26:45.510+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:45.510+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:45.839+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 1|2||5499a5d200df4929841901a9, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:45.840+0000 I SHARDING [conn2] collection benchdb1.COL-15 was previously unsharded, new metadata loaded with shard version 1|2||5499a5d200df4929841901a9 2014-12-23T17:26:45.840+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5d200df4929841901a9, took 0ms 2014-12-23T17:26:45.877+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:45.877+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d200df4929841901a9') } 2014-12-23T17:26:46.484+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5d59a8f659f34bfb604 2014-12-23T17:26:46.484+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:46-5499a5d69a8f659f34bfb605", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355606484), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:46.554+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 1|2||5499a5d200df4929841901a9, current metadata version is 1|2||5499a5d200df4929841901a9 2014-12-23T17:26:46.554+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-15 already up to date (shard version : 1|2||5499a5d200df4929841901a9, took 0ms) 2014-12-23T17:26:46.554+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5d200df4929841901a9 2014-12-23T17:26:46.555+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:46.558+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:46.560+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:46.564+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:46.564+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:46.565+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:46.565+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5d200df4929841901a9 2014-12-23T17:26:46.619+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:46.619+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5d200df4929841901a9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T17:26:46.923+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:46-5499a5d69a8f659f34bfb606", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355606923), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:46.974+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:46.974+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:46.974+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3031 2014-12-23T17:26:46.974+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:46.974+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:47.277+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:47.277+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:47-5499a5d79a8f659f34bfb607", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355607277), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 677, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 409, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:47.330+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d200df4929841901a9') } ntoreturn:1 keyUpdates:0 reslen:37 1453ms 2014-12-23T17:26:47.332+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:47.332+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d200df4929841901a9') } 2014-12-23T17:26:47.939+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5d79a8f659f34bfb608 2014-12-23T17:26:47.939+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:47-5499a5d79a8f659f34bfb609", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355607939), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:48.009+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 2|0||5499a5d200df4929841901a9, current metadata version is 2|0||5499a5d200df4929841901a9 2014-12-23T17:26:48.010+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-15 from shard version 2|0||5499a5d200df4929841901a9 to shard version 2|1||5499a5d200df4929841901a9 2014-12-23T17:26:48.010+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5d200df4929841901a9, took 0ms 2014-12-23T17:26:48.010+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5d200df4929841901a9 2014-12-23T17:26:48.010+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:48.012+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:48.015+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:48.019+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:48.028+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:48.044+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:48.044+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:48.044+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:48.044+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5d200df4929841901a9 2014-12-23T17:26:48.098+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:48.098+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5d200df4929841901a9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T17:26:48.403+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:48-5499a5d89a8f659f34bfb60a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355608403), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:48.468+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:48.468+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:48.468+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3039 2014-12-23T17:26:48.468+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:48.468+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:48.771+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:48.771+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:48-5499a5d89a8f659f34bfb60b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355608771), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 677, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:48.824+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d200df4929841901a9') } ntoreturn:1 keyUpdates:0 reslen:37 1491ms 2014-12-23T17:26:48.826+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5d200df4929841901a9') } 2014-12-23T17:26:49.433+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5d89a8f659f34bfb60c 2014-12-23T17:26:49.433+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 3|0||5499a5d200df4929841901a9, current metadata version is 3|0||5499a5d200df4929841901a9 2014-12-23T17:26:49.433+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-15 from shard version 3|0||5499a5d200df4929841901a9 to shard version 3|1||5499a5d200df4929841901a9 2014-12-23T17:26:49.434+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5d200df4929841901a9, took 0ms 2014-12-23T17:26:49.434+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5d200df4929841901a9 2014-12-23T17:26:49.738+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:49-5499a5d99a8f659f34bfb60d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355609738), what: "split", ns: "benchdb1.COL-15", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5d200df4929841901a9') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5d200df4929841901a9') } } } 2014-12-23T17:26:50.109+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:50.109+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5d200df4929841901a9') } ntoreturn:1 keyUpdates:0 reslen:37 1282ms 2014-12-23T17:26:52.372+0000 I INDEX [conn2] build index on: benchdb1.COL-16 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-16" } 2014-12-23T17:26:52.372+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:52.373+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:52.653+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 1|2||5499a5d900df4929841901ad, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:52.654+0000 I SHARDING [conn2] collection benchdb1.COL-16 was previously unsharded, new metadata loaded with shard version 1|2||5499a5d900df4929841901ad 2014-12-23T17:26:52.654+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5d900df4929841901ad, took 0ms 2014-12-23T17:26:52.697+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:52.697+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d900df4929841901ad') } 2014-12-23T17:26:52.992+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5dc9a8f659f34bfb60f 2014-12-23T17:26:52.992+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:52-5499a5dc9a8f659f34bfb610", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355612992), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:52.993+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:26:53.048+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 1|2||5499a5d900df4929841901ad, current metadata version is 1|2||5499a5d900df4929841901ad 2014-12-23T17:26:53.049+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-16 already up to date (shard version : 1|2||5499a5d900df4929841901ad, took 0ms) 2014-12-23T17:26:53.049+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5d900df4929841901ad 2014-12-23T17:26:53.049+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:53.052+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:53.054+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:53.058+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:53.058+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:53.059+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:53.059+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5d900df4929841901ad 2014-12-23T17:26:53.116+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:53.116+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5d900df4929841901ad through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T17:26:53.421+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:53-5499a5dd9a8f659f34bfb612", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355613421), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:53.421+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5dc9a8f659f34bfb611 2014-12-23T17:26:53.421+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:53-5499a5dd9a8f659f34bfb613", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355613421), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:26:53.483+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 2|5||5499a5bf00df49298419019b, current metadata version is 2|5||5499a5bf00df49298419019b 2014-12-23T17:26:53.484+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-12 from shard version 2|5||5499a5bf00df49298419019b to shard version 2|5||5499a5bf00df49298419019b 2014-12-23T17:26:53.484+0000 I SHARDING [conn4] collection version was loaded at version 2|7||5499a5bf00df49298419019b, took 0ms 2014-12-23T17:26:53.484+0000 I SHARDING [conn4] moveChunk request accepted at version 2|5||5499a5bf00df49298419019b 2014-12-23T17:26:53.484+0000 W SHARDING [conn4] moveChunk is already in progress from this shard 2014-12-23T17:26:53.514+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:53.514+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:53.514+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3060 2014-12-23T17:26:53.514+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:26:53.514+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:53.514+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019] 2014-12-23T17:26:53.515+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019] 2014-12-23T17:26:53.515+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019] 2014-12-23T17:26:53.717+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:53.717+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:53-5499a5dd9a8f659f34bfb614", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355613717), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, step 2 of 6: 491, to: "shard0001", from: "shard0002", note: "aborted", errmsg: "moveChunk is already in progress from this shard" } } 2014-12-23T17:26:53.767+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } ntoreturn:1 keyUpdates:0 reslen:98 774ms 2014-12-23T17:26:53.812+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:53.812+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:53-5499a5dd9a8f659f34bfb615", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355613812), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 351, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 455, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:26:53.860+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d900df4929841901ad') } ntoreturn:1 keyUpdates:0 reslen:37 1163ms 2014-12-23T17:26:53.861+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:53.862+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d900df4929841901ad') } 2014-12-23T17:26:54.418+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5dd9a8f659f34bfb616 2014-12-23T17:26:54.418+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:54-5499a5de9a8f659f34bfb617", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355614418), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:26:54.481+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 2|0||5499a5d900df4929841901ad, current metadata version is 2|0||5499a5d900df4929841901ad 2014-12-23T17:26:54.482+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-16 from shard version 2|0||5499a5d900df4929841901ad to shard version 2|1||5499a5d900df4929841901ad 2014-12-23T17:26:54.482+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5d900df4929841901ad, took 0ms 2014-12-23T17:26:54.482+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5d900df4929841901ad 2014-12-23T17:26:54.482+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:26:54.485+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:54.487+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:54.491+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:54.500+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:54.516+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:26:54.516+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:26:54.516+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:26:54.517+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5d900df4929841901ad 2014-12-23T17:26:54.580+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:26:54.580+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5d900df4929841901ad through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T17:26:54.884+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:54-5499a5de9a8f659f34bfb618", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355614884), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:26:54.942+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:54.942+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:26:54.942+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3069 2014-12-23T17:26:54.942+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:26:54.942+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:26:55.245+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:55.245+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:55-5499a5df9a8f659f34bfb619", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355615245), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 620, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 425, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:26:55.303+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5d900df4929841901ad') } ntoreturn:1 keyUpdates:0 reslen:37 1442ms 2014-12-23T17:26:55.305+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5d900df4929841901ad') } 2014-12-23T17:26:55.911+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5df9a8f659f34bfb61a 2014-12-23T17:26:55.911+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 3|0||5499a5d900df4929841901ad, current metadata version is 3|0||5499a5d900df4929841901ad 2014-12-23T17:26:55.912+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-16 from shard version 3|0||5499a5d900df4929841901ad to shard version 3|1||5499a5d900df4929841901ad 2014-12-23T17:26:55.912+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5d900df4929841901ad, took 0ms 2014-12-23T17:26:55.912+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5d900df4929841901ad 2014-12-23T17:26:56.217+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:56-5499a5e09a8f659f34bfb61b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355616217), what: "split", ns: "benchdb1.COL-16", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5d900df4929841901ad') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5d900df4929841901ad') } } } 2014-12-23T17:26:56.588+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:26:56.588+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5d900df4929841901ad') } ntoreturn:1 keyUpdates:0 reslen:37 1282ms 2014-12-23T17:26:59.173+0000 I INDEX [conn2] build index on: benchdb1.COL-17 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-17" } 2014-12-23T17:26:59.173+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:26:59.174+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:26:59.450+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-17 with requested shard version 1|2||5499a5e000df4929841901b1, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:26:59.451+0000 I SHARDING [conn2] collection benchdb1.COL-17 was previously unsharded, new metadata loaded with shard version 1|2||5499a5e000df4929841901b1 2014-12-23T17:26:59.451+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5e000df4929841901b1, took 0ms 2014-12-23T17:26:59.497+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:26:59.497+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e000df4929841901b1') } 2014-12-23T17:26:59.956+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5e39a8f659f34bfb61d 2014-12-23T17:26:59.956+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:26:59-5499a5e39a8f659f34bfb61e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355619956), what: "moveChunk.start", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:00.018+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 1|2||5499a5e000df4929841901b1, current metadata version is 1|2||5499a5e000df4929841901b1 2014-12-23T17:27:00.019+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-17 already up to date (shard version : 1|2||5499a5e000df4929841901b1, took 0ms) 2014-12-23T17:27:00.019+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5e000df4929841901b1 2014-12-23T17:27:00.019+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:00.022+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:00.025+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:00.029+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:00.029+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:00.029+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:00.029+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5e000df4929841901b1 2014-12-23T17:27:00.082+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:00.082+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5e000df4929841901b1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-17' 2014-12-23T17:27:00.386+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:00-5499a5e49a8f659f34bfb61f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355620386), what: "moveChunk.commit", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:00.439+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:00.439+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:00.439+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-17 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3088 2014-12-23T17:27:00.439+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-17 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:00.439+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:00.725+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:00.725+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:00-5499a5e49a8f659f34bfb620", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355620725), what: "moveChunk.from", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 522, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 410, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:00.777+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e000df4929841901b1') } ntoreturn:1 keyUpdates:0 reslen:37 1280ms 2014-12-23T17:27:00.778+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:00.779+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e000df4929841901b1') } 2014-12-23T17:27:01.385+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5e49a8f659f34bfb621 2014-12-23T17:27:01.385+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:01-5499a5e59a8f659f34bfb622", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355621385), what: "moveChunk.start", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:01.449+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 2|0||5499a5e000df4929841901b1, current metadata version is 2|0||5499a5e000df4929841901b1 2014-12-23T17:27:01.450+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-17 from shard version 2|0||5499a5e000df4929841901b1 to shard version 2|1||5499a5e000df4929841901b1 2014-12-23T17:27:01.450+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5e000df4929841901b1, took 0ms 2014-12-23T17:27:01.450+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5e000df4929841901b1 2014-12-23T17:27:01.450+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:01.452+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:01.455+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:01.459+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:01.468+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:01.484+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:01.484+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:01.484+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:01.485+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5e000df4929841901b1 2014-12-23T17:27:01.537+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:01.537+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5e000df4929841901b1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-17' 2014-12-23T17:27:01.842+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:01-5499a5e59a8f659f34bfb623", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355621842), what: "moveChunk.commit", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:01.902+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:01.902+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:01.902+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-17 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3096 2014-12-23T17:27:01.902+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-17 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:01.902+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:02.208+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:02.208+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:02-5499a5e69a8f659f34bfb624", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355622208), what: "moveChunk.from", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 417, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:02.264+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e000df4929841901b1') } ntoreturn:1 keyUpdates:0 reslen:37 1485ms 2014-12-23T17:27:02.266+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-17", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-17-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5e000df4929841901b1') } 2014-12-23T17:27:02.874+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5e69a8f659f34bfb625 2014-12-23T17:27:02.874+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 3|0||5499a5e000df4929841901b1, current metadata version is 3|0||5499a5e000df4929841901b1 2014-12-23T17:27:02.875+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-17 from shard version 3|0||5499a5e000df4929841901b1 to shard version 3|1||5499a5e000df4929841901b1 2014-12-23T17:27:02.875+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5e000df4929841901b1, took 0ms 2014-12-23T17:27:02.875+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5e000df4929841901b1 2014-12-23T17:27:03.180+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:03-5499a5e79a8f659f34bfb626", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355623180), what: "split", ns: "benchdb1.COL-17", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5e000df4929841901b1') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5e000df4929841901b1') } } } 2014-12-23T17:27:03.551+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:03.552+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-17", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-17-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5e000df4929841901b1') } ntoreturn:1 keyUpdates:0 reslen:37 1285ms 2014-12-23T17:27:05.187+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:27:05.529+0000 I INDEX [conn2] build index on: benchdb1.COL-18 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-18" } 2014-12-23T17:27:05.529+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:05.530+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:05.697+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5e99a8f659f34bfb627 2014-12-23T17:27:05.697+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:05-5499a5e99a8f659f34bfb629", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355625697), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:05.782+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 2|5||5499a5bf00df49298419019b, current metadata version is 2|7||5499a5bf00df49298419019b 2014-12-23T17:27:05.783+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-12 already up to date (shard version : 2|5||5499a5bf00df49298419019b, took 0ms) 2014-12-23T17:27:05.783+0000 I SHARDING [conn1] moveChunk request accepted at version 2|5||5499a5bf00df49298419019b 2014-12-23T17:27:05.783+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:05.786+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:05.788+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:05.792+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:05.792+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:05.793+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:05.793+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5bf00df49298419019b 2014-12-23T17:27:05.858+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:05.858+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5bf00df49298419019b through { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T17:27:05.921+0000 I SHARDING [conn2] waiting till out of critical section 2014-12-23T17:27:06.124+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:06-5499a5ea9a8f659f34bfb62a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355626124), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:06.124+0000 I QUERY [conn2] command admin.$cmd command: setShardVersion { setShardVersion: "benchdb1.COL-18", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", shard: "shard0002", shardHost: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", version: Timestamp 1000|2, versionEpoch: ObjectId('5499a5e600df4929841901b5') } ntoreturn:1 keyUpdates:0 reslen:187 203ms 2014-12-23T17:27:06.125+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-18 with requested shard version 1|2||5499a5e600df4929841901b5, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:06.126+0000 I SHARDING [conn2] collection benchdb1.COL-18 was previously unsharded, new metadata loaded with shard version 1|2||5499a5e600df4929841901b5 2014-12-23T17:27:06.126+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5e600df4929841901b5, took 0ms 2014-12-23T17:27:06.178+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:06.178+0000 I SHARDING [conn1] forking for cleanup of chunk data 2014-12-23T17:27:06.178+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-12 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 }, with opId: 3 2014-12-23T17:27:06.178+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 } 2014-12-23T17:27:06.178+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:06.208+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:27:06.208+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e600df4929841901b5') } 2014-12-23T17:27:06.410+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:06.410+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:06-5499a5ea9a8f659f34bfb62c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355626410), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, step 2 of 6: 595, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 385, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:06.410+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019] 2014-12-23T17:27:06.411+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019] 2014-12-23T17:27:06.411+0000 I NETWORK [conn1] SyncClusterConnection connecting to [ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019] 2014-12-23T17:27:06.468+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } ntoreturn:1 keyUpdates:0 reslen:37 1281ms 2014-12-23T17:27:06.718+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5ea9a8f659f34bfb62b 2014-12-23T17:27:06.718+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:06-5499a5ea9a8f659f34bfb62d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355626718), what: "moveChunk.start", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:06.792+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 1|2||5499a5e600df4929841901b5, current metadata version is 1|2||5499a5e600df4929841901b5 2014-12-23T17:27:06.793+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-18 already up to date (shard version : 1|2||5499a5e600df4929841901b5, took 0ms) 2014-12-23T17:27:06.793+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a5e600df4929841901b5 2014-12-23T17:27:06.794+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:27:06.796+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:06.799+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:06.803+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:06.803+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:27:06.803+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:27:06.803+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a5e600df4929841901b5 2014-12-23T17:27:06.843+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:27:06 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:27:06.869+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:06.869+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a5e600df4929841901b5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-18' 2014-12-23T17:27:07.174+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:07-5499a5eb9a8f659f34bfb62e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355627174), what: "moveChunk.commit", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:07.234+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:07.234+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:27:07.234+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-18 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3125 2014-12-23T17:27:07.234+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-18 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:07.234+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:07.537+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:07.537+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:07-5499a5eb9a8f659f34bfb62f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355627537), what: "moveChunk.from", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 585, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 430, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:07.590+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e600df4929841901b5') } ntoreturn:1 keyUpdates:0 reslen:37 1382ms 2014-12-23T17:27:07.591+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:27:07.592+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e600df4929841901b5') } 2014-12-23T17:27:08.135+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5eb9a8f659f34bfb630 2014-12-23T17:27:08.135+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:08-5499a5ec9a8f659f34bfb631", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355628135), what: "moveChunk.start", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:08.165+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 2|0||5499a5e600df4929841901b5, current metadata version is 2|0||5499a5e600df4929841901b5 2014-12-23T17:27:08.166+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-18 from shard version 2|0||5499a5e600df4929841901b5 to shard version 2|1||5499a5e600df4929841901b5 2014-12-23T17:27:08.166+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a5e600df4929841901b5, took 0ms 2014-12-23T17:27:08.166+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a5e600df4929841901b5 2014-12-23T17:27:08.166+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:27:08.169+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:08.171+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:08.176+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:08.184+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:08.200+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:08.200+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:27:08.201+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:27:08.201+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a5e600df4929841901b5 2014-12-23T17:27:08.271+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:08.271+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a5e600df4929841901b5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-18' 2014-12-23T17:27:08.473+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:08-5499a5ec9a8f659f34bfb632", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355628473), what: "moveChunk.commit", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:08.530+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:08.530+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:27:08.530+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-18 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3133 2014-12-23T17:27:08.531+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-18 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:08.531+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:08.704+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:08.704+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:08-5499a5ec9a8f659f34bfb633", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355628704), what: "moveChunk.from", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 574, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 330, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:08.765+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5e600df4929841901b5') } ntoreturn:1 keyUpdates:0 reslen:37 1173ms 2014-12-23T17:27:08.766+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-18", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-18-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5e600df4929841901b5') } 2014-12-23T17:27:08.981+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } 2014-12-23T17:27:09.081+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5ec9a8f659f34bfb634 2014-12-23T17:27:09.081+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 3|0||5499a5e600df4929841901b5, current metadata version is 3|0||5499a5e600df4929841901b5 2014-12-23T17:27:09.081+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-18 from shard version 3|0||5499a5e600df4929841901b5 to shard version 3|1||5499a5e600df4929841901b5 2014-12-23T17:27:09.081+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a5e600df4929841901b5, took 0ms 2014-12-23T17:27:09.081+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a5e600df4929841901b5 2014-12-23T17:27:09.338+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:09-5499a5ed9a8f659f34bfb636", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355629338), what: "split", ns: "benchdb1.COL-18", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5e600df4929841901b5') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5e600df4929841901b5') } } } 2014-12-23T17:27:09.461+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5ec9a8f659f34bfb635 2014-12-23T17:27:09.461+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:09-5499a5ed9a8f659f34bfb637", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355629461), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:09.508+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 3|0||5499a5bf00df49298419019b, current metadata version is 3|0||5499a5bf00df49298419019b 2014-12-23T17:27:09.508+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-12 from shard version 3|0||5499a5bf00df49298419019b to shard version 3|1||5499a5bf00df49298419019b 2014-12-23T17:27:09.508+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5bf00df49298419019b, took 0ms 2014-12-23T17:27:09.508+0000 I SHARDING [conn1] moveChunk request accepted at version 3|1||5499a5bf00df49298419019b 2014-12-23T17:27:09.508+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:09.511+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:09.514+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:09.514+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:09.514+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:09.514+0000 I SHARDING [conn1] moveChunk setting version to: 4|0||5499a5bf00df49298419019b 2014-12-23T17:27:09.569+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:09.569+0000 I SHARDING [conn1] moveChunk updating self version to: 4|1||5499a5bf00df49298419019b through { shardkey: -3074457345618258602 } -> { shardkey: 0 } for collection 'benchdb1.COL-12' 2014-12-23T17:27:09.670+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:09.670+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-18", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-18-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5e600df4929841901b5') } ntoreturn:1 keyUpdates:0 reslen:37 903ms 2014-12-23T17:27:09.881+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:09-5499a5ed9a8f659f34bfb638", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355629881), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:09.946+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:09.946+0000 I SHARDING [conn1] forking for cleanup of chunk data 2014-12-23T17:27:09.946+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-12 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T17:27:09.946+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } 2014-12-23T17:27:09.946+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:10.249+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:10.249+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:10-5499a5ee9a8f659f34bfb639", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355630249), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 527, step 3 of 6: 1, step 4 of 6: 3, step 5 of 6: 431, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:10.301+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a5bf00df49298419019b') } ntoreturn:1 keyUpdates:0 reslen:37 1320ms 2014-12-23T17:27:11.946+0000 I INDEX [conn2] build index on: benchdb1.COL-19 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-19" } 2014-12-23T17:27:11.946+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:11.946+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:12.330+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-19 with requested shard version 1|2||5499a5ec00df4929841901bc, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:12.331+0000 I SHARDING [conn2] collection benchdb1.COL-19 was previously unsharded, new metadata loaded with shard version 1|2||5499a5ec00df4929841901bc 2014-12-23T17:27:12.331+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5ec00df4929841901bc, took 0ms 2014-12-23T17:27:12.380+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:12.381+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5ec00df4929841901bc') } 2014-12-23T17:27:12.710+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5f09a8f659f34bfb63b 2014-12-23T17:27:12.710+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:12-5499a5f09a8f659f34bfb63c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355632710), what: "moveChunk.start", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:12.773+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 1|2||5499a5ec00df4929841901bc, current metadata version is 1|2||5499a5ec00df4929841901bc 2014-12-23T17:27:12.774+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-19 already up to date (shard version : 1|2||5499a5ec00df4929841901bc, took 0ms) 2014-12-23T17:27:12.774+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5ec00df4929841901bc 2014-12-23T17:27:12.774+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:12.777+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:12.780+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:12.784+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:12.784+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:12.784+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:12.784+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5ec00df4929841901bc 2014-12-23T17:27:12.836+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:12.836+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5ec00df4929841901bc through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-19' 2014-12-23T17:27:13.041+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:13-5499a5f19a8f659f34bfb63d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355633041), what: "moveChunk.commit", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:13.115+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:13.115+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:13.115+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-19 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3163 2014-12-23T17:27:13.115+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-19 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:13.115+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:13.335+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:13.335+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:13-5499a5f19a8f659f34bfb63e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355633335), what: "moveChunk.from", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 393, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 331, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:13.389+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5ec00df4929841901bc') } ntoreturn:1 keyUpdates:0 reslen:37 1008ms 2014-12-23T17:27:13.391+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:13.391+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5ec00df4929841901bc') } 2014-12-23T17:27:13.981+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5f19a8f659f34bfb63f 2014-12-23T17:27:13.981+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:13-5499a5f19a8f659f34bfb640", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355633981), what: "moveChunk.start", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:14.060+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 2|0||5499a5ec00df4929841901bc, current metadata version is 2|0||5499a5ec00df4929841901bc 2014-12-23T17:27:14.060+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-19 from shard version 2|0||5499a5ec00df4929841901bc to shard version 2|1||5499a5ec00df4929841901bc 2014-12-23T17:27:14.060+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5ec00df4929841901bc, took 0ms 2014-12-23T17:27:14.060+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5ec00df4929841901bc 2014-12-23T17:27:14.061+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:14.063+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:14.065+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:14.070+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:14.078+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:14.095+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:14.095+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:14.095+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:14.095+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5ec00df4929841901bc 2014-12-23T17:27:14.154+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:14.154+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5ec00df4929841901bc through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-19' 2014-12-23T17:27:14.418+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:14-5499a5f29a8f659f34bfb641", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355634418), what: "moveChunk.commit", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:14.480+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:14.480+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:14.481+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-19 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3171 2014-12-23T17:27:14.481+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-19 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:14.481+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:14.784+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:14.784+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:14-5499a5f29a8f659f34bfb642", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355634784), what: "moveChunk.from", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 385, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:14.840+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5ec00df4929841901bc') } ntoreturn:1 keyUpdates:0 reslen:37 1449ms 2014-12-23T17:27:14.842+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-19", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-19-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5ec00df4929841901bc') } 2014-12-23T17:27:15.449+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5f29a8f659f34bfb643 2014-12-23T17:27:15.449+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 3|0||5499a5ec00df4929841901bc, current metadata version is 3|0||5499a5ec00df4929841901bc 2014-12-23T17:27:15.450+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-19 from shard version 3|0||5499a5ec00df4929841901bc to shard version 3|1||5499a5ec00df4929841901bc 2014-12-23T17:27:15.450+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5ec00df4929841901bc, took 0ms 2014-12-23T17:27:15.450+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5ec00df4929841901bc 2014-12-23T17:27:15.754+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:15-5499a5f39a8f659f34bfb644", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355635754), what: "split", ns: "benchdb1.COL-19", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5ec00df4929841901bc') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5ec00df4929841901bc') } } } 2014-12-23T17:27:16.126+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:16.126+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-19", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-19-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5ec00df4929841901bc') } ntoreturn:1 keyUpdates:0 reslen:37 1283ms 2014-12-23T17:27:18.709+0000 I INDEX [conn2] build index on: benchdb1.COL-20 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-20" } 2014-12-23T17:27:18.709+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:18.710+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:18.984+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-20 with requested shard version 1|2||5499a5f300df4929841901c0, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:18.985+0000 I SHARDING [conn2] collection benchdb1.COL-20 was previously unsharded, new metadata loaded with shard version 1|2||5499a5f300df4929841901c0 2014-12-23T17:27:18.985+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5f300df4929841901c0, took 0ms 2014-12-23T17:27:19.028+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:19.028+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f300df4929841901c0') } 2014-12-23T17:27:19.635+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5f79a8f659f34bfb646 2014-12-23T17:27:19.635+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:19-5499a5f79a8f659f34bfb647", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355639635), what: "moveChunk.start", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:19.698+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 1|2||5499a5f300df4929841901c0, current metadata version is 1|2||5499a5f300df4929841901c0 2014-12-23T17:27:19.698+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-20 already up to date (shard version : 1|2||5499a5f300df4929841901c0, took 0ms) 2014-12-23T17:27:19.699+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5f300df4929841901c0 2014-12-23T17:27:19.699+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:19.702+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:19.704+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:19.708+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:19.708+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:19.709+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:19.709+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5f300df4929841901c0 2014-12-23T17:27:19.760+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:19.760+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5f300df4929841901c0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-20' 2014-12-23T17:27:20.065+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:20-5499a5f89a8f659f34bfb648", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355640065), what: "moveChunk.commit", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:20.113+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:20.113+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:20.113+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-20 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3190 2014-12-23T17:27:20.113+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-20 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:20.113+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:20.416+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:20.416+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:20-5499a5f89a8f659f34bfb649", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355640416), what: "moveChunk.from", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 404, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:20.473+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f300df4929841901c0') } ntoreturn:1 keyUpdates:0 reslen:37 1445ms 2014-12-23T17:27:20.475+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:20.475+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f300df4929841901c0') } 2014-12-23T17:27:21.081+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5f89a8f659f34bfb64a 2014-12-23T17:27:21.081+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:21-5499a5f99a8f659f34bfb64b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355641081), what: "moveChunk.start", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:21.147+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 2|0||5499a5f300df4929841901c0, current metadata version is 2|0||5499a5f300df4929841901c0 2014-12-23T17:27:21.148+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-20 from shard version 2|0||5499a5f300df4929841901c0 to shard version 2|1||5499a5f300df4929841901c0 2014-12-23T17:27:21.148+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5f300df4929841901c0, took 0ms 2014-12-23T17:27:21.148+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5f300df4929841901c0 2014-12-23T17:27:21.148+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:21.150+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:21.153+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:21.157+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:21.166+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:21.182+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:21.182+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:21.183+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:21.183+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5f300df4929841901c0 2014-12-23T17:27:21.241+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:21.241+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5f300df4929841901c0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-20' 2014-12-23T17:27:21.546+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:21-5499a5f99a8f659f34bfb64c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355641546), what: "moveChunk.commit", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:21.611+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:21.611+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:21.611+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-20 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3198 2014-12-23T17:27:21.611+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-20 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:21.611+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:21.914+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:21.914+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:21-5499a5f99a8f659f34bfb64d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355641914), what: "moveChunk.from", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 428, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:21.970+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f300df4929841901c0') } ntoreturn:1 keyUpdates:0 reslen:37 1495ms 2014-12-23T17:27:21.972+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-20", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-20-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5f300df4929841901c0') } 2014-12-23T17:27:22.579+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5f99a8f659f34bfb64e 2014-12-23T17:27:22.579+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 3|0||5499a5f300df4929841901c0, current metadata version is 3|0||5499a5f300df4929841901c0 2014-12-23T17:27:22.580+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-20 from shard version 3|0||5499a5f300df4929841901c0 to shard version 3|1||5499a5f300df4929841901c0 2014-12-23T17:27:22.580+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5f300df4929841901c0, took 0ms 2014-12-23T17:27:22.580+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5f300df4929841901c0 2014-12-23T17:27:22.885+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:22-5499a5fa9a8f659f34bfb64f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355642885), what: "split", ns: "benchdb1.COL-20", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5f300df4929841901c0') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5f300df4929841901c0') } } } 2014-12-23T17:27:23.271+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:23.271+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-20", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-20-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5f300df4929841901c0') } ntoreturn:1 keyUpdates:0 reslen:37 1298ms 2014-12-23T17:27:24.934+0000 I INDEX [conn2] build index on: benchdb1.COL-21 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-21" } 2014-12-23T17:27:24.934+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:24.934+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:25.239+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-21 with requested shard version 1|2||5499a5f900df4929841901c5, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:25.240+0000 I SHARDING [conn2] collection benchdb1.COL-21 was previously unsharded, new metadata loaded with shard version 1|2||5499a5f900df4929841901c5 2014-12-23T17:27:25.240+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a5f900df4929841901c5, took 0ms 2014-12-23T17:27:25.281+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:25.281+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f900df4929841901c5') } 2014-12-23T17:27:25.888+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5fd9a8f659f34bfb651 2014-12-23T17:27:25.888+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:25-5499a5fd9a8f659f34bfb652", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355645888), what: "moveChunk.start", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:25.944+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 1|2||5499a5f900df4929841901c5, current metadata version is 1|2||5499a5f900df4929841901c5 2014-12-23T17:27:25.945+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-21 already up to date (shard version : 1|2||5499a5f900df4929841901c5, took 0ms) 2014-12-23T17:27:25.945+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a5f900df4929841901c5 2014-12-23T17:27:25.945+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:25.948+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:25.951+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:25.955+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:25.955+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:25.955+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:25.955+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a5f900df4929841901c5 2014-12-23T17:27:26.013+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:26.013+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a5f900df4929841901c5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-21' 2014-12-23T17:27:26.318+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:26-5499a5fe9a8f659f34bfb653", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355646318), what: "moveChunk.commit", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:26.369+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:26.369+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:26.369+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-21 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3219 2014-12-23T17:27:26.369+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-21 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:26.369+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:26.672+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:26.672+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:26-5499a5fe9a8f659f34bfb654", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355646672), what: "moveChunk.from", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:26.731+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f900df4929841901c5') } ntoreturn:1 keyUpdates:0 reslen:37 1449ms 2014-12-23T17:27:26.732+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:26.733+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f900df4929841901c5') } 2014-12-23T17:27:27.339+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a5fe9a8f659f34bfb655 2014-12-23T17:27:27.339+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:27-5499a5ff9a8f659f34bfb656", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355647339), what: "moveChunk.start", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:27.404+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 2|0||5499a5f900df4929841901c5, current metadata version is 2|0||5499a5f900df4929841901c5 2014-12-23T17:27:27.405+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-21 from shard version 2|0||5499a5f900df4929841901c5 to shard version 2|1||5499a5f900df4929841901c5 2014-12-23T17:27:27.405+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a5f900df4929841901c5, took 0ms 2014-12-23T17:27:27.405+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a5f900df4929841901c5 2014-12-23T17:27:27.405+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:27.407+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:27.409+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:27.414+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:27.422+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:27.439+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:27.439+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:27.439+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:27.439+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a5f900df4929841901c5 2014-12-23T17:27:27.491+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:27.491+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a5f900df4929841901c5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-21' 2014-12-23T17:27:27.795+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:27-5499a5ff9a8f659f34bfb657", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355647795), what: "moveChunk.commit", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:27.857+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:27.857+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:27.857+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-21 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3227 2014-12-23T17:27:27.857+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-21 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:27.857+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:28.160+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:28.160+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:28-5499a6009a8f659f34bfb658", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355648160), what: "moveChunk.from", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 418, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:28.221+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a5f900df4929841901c5') } ntoreturn:1 keyUpdates:0 reslen:37 1488ms 2014-12-23T17:27:28.223+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-21", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-21-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5f900df4929841901c5') } 2014-12-23T17:27:28.829+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6009a8f659f34bfb659 2014-12-23T17:27:28.829+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 3|0||5499a5f900df4929841901c5, current metadata version is 3|0||5499a5f900df4929841901c5 2014-12-23T17:27:28.830+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-21 from shard version 3|0||5499a5f900df4929841901c5 to shard version 3|1||5499a5f900df4929841901c5 2014-12-23T17:27:28.830+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a5f900df4929841901c5, took 0ms 2014-12-23T17:27:28.830+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a5f900df4929841901c5 2014-12-23T17:27:29.134+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:29-5499a6019a8f659f34bfb65a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355649134), what: "split", ns: "benchdb1.COL-21", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a5f900df4929841901c5') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a5f900df4929841901c5') } } } 2014-12-23T17:27:29.507+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:29.507+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-21", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-21-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a5f900df4929841901c5') } ntoreturn:1 keyUpdates:0 reslen:37 1284ms 2014-12-23T17:27:32.085+0000 I INDEX [conn2] build index on: benchdb1.COL-22 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-22" } 2014-12-23T17:27:32.085+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:32.086+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:32.414+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-22 with requested shard version 1|2||5499a60100df4929841901c8, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:32.415+0000 I SHARDING [conn2] collection benchdb1.COL-22 was previously unsharded, new metadata loaded with shard version 1|2||5499a60100df4929841901c8 2014-12-23T17:27:32.415+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a60100df4929841901c8, took 0ms 2014-12-23T17:27:32.456+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:32.457+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60100df4929841901c8') } 2014-12-23T17:27:33.063+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6049a8f659f34bfb65c 2014-12-23T17:27:33.064+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:33-5499a6059a8f659f34bfb65d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355653064), what: "moveChunk.start", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:33.127+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 1|2||5499a60100df4929841901c8, current metadata version is 1|2||5499a60100df4929841901c8 2014-12-23T17:27:33.128+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-22 already up to date (shard version : 1|2||5499a60100df4929841901c8, took 0ms) 2014-12-23T17:27:33.128+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a60100df4929841901c8 2014-12-23T17:27:33.128+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:33.131+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:33.133+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:33.138+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:33.138+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:33.138+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:33.138+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a60100df4929841901c8 2014-12-23T17:27:33.191+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:33.191+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a60100df4929841901c8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-22' 2014-12-23T17:27:33.496+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:33-5499a6059a8f659f34bfb65e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355653496), what: "moveChunk.commit", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:33.579+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:33.579+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:33.579+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-22 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3245 2014-12-23T17:27:33.579+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-22 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:33.579+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:33.882+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:33.882+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:33-5499a6059a8f659f34bfb65f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355653882), what: "moveChunk.from", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 441, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:33.939+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60100df4929841901c8') } ntoreturn:1 keyUpdates:0 reslen:37 1482ms 2014-12-23T17:27:33.941+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:33.941+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60100df4929841901c8') } 2014-12-23T17:27:34.548+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6059a8f659f34bfb660 2014-12-23T17:27:34.548+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:34-5499a6069a8f659f34bfb661", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355654548), what: "moveChunk.start", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:34.603+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 2|0||5499a60100df4929841901c8, current metadata version is 2|0||5499a60100df4929841901c8 2014-12-23T17:27:34.604+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-22 from shard version 2|0||5499a60100df4929841901c8 to shard version 2|1||5499a60100df4929841901c8 2014-12-23T17:27:34.604+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a60100df4929841901c8, took 0ms 2014-12-23T17:27:34.604+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a60100df4929841901c8 2014-12-23T17:27:34.604+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:34.606+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:34.608+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:34.613+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:34.621+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:34.638+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:34.638+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:34.638+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:34.638+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a60100df4929841901c8 2014-12-23T17:27:34.692+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:34.692+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a60100df4929841901c8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-22' 2014-12-23T17:27:34.997+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:34-5499a6069a8f659f34bfb662", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355654997), what: "moveChunk.commit", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:35.062+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:35.062+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:35.062+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-22 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3253 2014-12-23T17:27:35.062+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-22 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:35.062+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:35.365+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:35.365+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:35-5499a6079a8f659f34bfb663", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355655365), what: "moveChunk.from", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:35.428+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60100df4929841901c8') } ntoreturn:1 keyUpdates:0 reslen:37 1487ms 2014-12-23T17:27:35.430+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-22", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-22-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a60100df4929841901c8') } 2014-12-23T17:27:35.811+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6079a8f659f34bfb664 2014-12-23T17:27:35.811+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 3|0||5499a60100df4929841901c8, current metadata version is 3|0||5499a60100df4929841901c8 2014-12-23T17:27:35.812+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-22 from shard version 3|0||5499a60100df4929841901c8 to shard version 3|1||5499a60100df4929841901c8 2014-12-23T17:27:35.812+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a60100df4929841901c8, took 0ms 2014-12-23T17:27:35.812+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a60100df4929841901c8 2014-12-23T17:27:36.010+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:36-5499a6089a8f659f34bfb665", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355656010), what: "split", ns: "benchdb1.COL-22", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a60100df4929841901c8') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a60100df4929841901c8') } } } 2014-12-23T17:27:36.240+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:36.240+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-22", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-22-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a60100df4929841901c8') } ntoreturn:1 keyUpdates:0 reslen:37 810ms 2014-12-23T17:27:37.346+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:27:36 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:27:38.546+0000 I INDEX [conn2] build index on: benchdb1.COL-23 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-23" } 2014-12-23T17:27:38.546+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:38.546+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:38.837+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-23 with requested shard version 1|2||5499a60700df4929841901cd, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:38.837+0000 I SHARDING [conn2] collection benchdb1.COL-23 was previously unsharded, new metadata loaded with shard version 1|2||5499a60700df4929841901cd 2014-12-23T17:27:38.838+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a60700df4929841901cd, took 0ms 2014-12-23T17:27:38.880+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:38.881+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60700df4929841901cd') } 2014-12-23T17:27:39.438+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a60a9a8f659f34bfb667 2014-12-23T17:27:39.438+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:39-5499a60b9a8f659f34bfb668", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355659438), what: "moveChunk.start", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:39.511+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 1|2||5499a60700df4929841901cd, current metadata version is 1|2||5499a60700df4929841901cd 2014-12-23T17:27:39.511+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-23 already up to date (shard version : 1|2||5499a60700df4929841901cd, took 0ms) 2014-12-23T17:27:39.511+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a60700df4929841901cd 2014-12-23T17:27:39.512+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:39.514+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:39.517+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:39.521+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:39.521+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:39.521+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:39.521+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a60700df4929841901cd 2014-12-23T17:27:39.585+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:39.585+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a60700df4929841901cd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-23' 2014-12-23T17:27:39.890+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:39-5499a60b9a8f659f34bfb669", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355659890), what: "moveChunk.commit", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:39.985+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:39.985+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:39.986+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-23 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3274 2014-12-23T17:27:39.986+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-23 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:39.986+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:40.289+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:40.289+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:40-5499a60c9a8f659f34bfb66a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355660289), what: "moveChunk.from", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 630, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 464, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:40.342+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60700df4929841901cd') } ntoreturn:1 keyUpdates:0 reslen:37 1461ms 2014-12-23T17:27:40.343+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:40.344+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60700df4929841901cd') } 2014-12-23T17:27:40.951+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a60c9a8f659f34bfb66b 2014-12-23T17:27:40.951+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:40-5499a60c9a8f659f34bfb66c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355660951), what: "moveChunk.start", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:41.011+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 2|0||5499a60700df4929841901cd, current metadata version is 2|0||5499a60700df4929841901cd 2014-12-23T17:27:41.012+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-23 from shard version 2|0||5499a60700df4929841901cd to shard version 2|1||5499a60700df4929841901cd 2014-12-23T17:27:41.012+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a60700df4929841901cd, took 0ms 2014-12-23T17:27:41.012+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a60700df4929841901cd 2014-12-23T17:27:41.012+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:41.015+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:41.017+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:41.022+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:41.030+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:41.046+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:41.047+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:41.047+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:41.047+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a60700df4929841901cd 2014-12-23T17:27:41.101+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:41.101+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a60700df4929841901cd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-23' 2014-12-23T17:27:41.406+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:41-5499a60d9a8f659f34bfb66d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355661406), what: "moveChunk.commit", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:41.465+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:41.465+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:41.465+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-23 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3282 2014-12-23T17:27:41.465+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-23 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:41.465+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:41.768+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:41.769+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:41-5499a60d9a8f659f34bfb66e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355661769), what: "moveChunk.from", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 418, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:41.834+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60700df4929841901cd') } ntoreturn:1 keyUpdates:0 reslen:37 1490ms 2014-12-23T17:27:41.836+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-23", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-23-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a60700df4929841901cd') } 2014-12-23T17:27:42.443+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a60d9a8f659f34bfb66f 2014-12-23T17:27:42.443+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 3|0||5499a60700df4929841901cd, current metadata version is 3|0||5499a60700df4929841901cd 2014-12-23T17:27:42.444+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-23 from shard version 3|0||5499a60700df4929841901cd to shard version 3|1||5499a60700df4929841901cd 2014-12-23T17:27:42.444+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a60700df4929841901cd, took 0ms 2014-12-23T17:27:42.444+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a60700df4929841901cd 2014-12-23T17:27:42.749+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:42-5499a60e9a8f659f34bfb670", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355662749), what: "split", ns: "benchdb1.COL-23", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a60700df4929841901cd') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a60700df4929841901cd') } } } 2014-12-23T17:27:43.122+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:43.123+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-23", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-23-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a60700df4929841901cd') } ntoreturn:1 keyUpdates:0 reslen:37 1286ms 2014-12-23T17:27:45.618+0000 I INDEX [conn2] build index on: benchdb1.COL-24 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-24" } 2014-12-23T17:27:45.618+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:45.619+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:45.902+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-24 with requested shard version 1|2||5499a60e00df4929841901d0, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:45.902+0000 I SHARDING [conn2] collection benchdb1.COL-24 was previously unsharded, new metadata loaded with shard version 1|2||5499a60e00df4929841901d0 2014-12-23T17:27:45.902+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a60e00df4929841901d0, took 0ms 2014-12-23T17:27:45.943+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:45.943+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60e00df4929841901d0') } 2014-12-23T17:27:46.550+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6119a8f659f34bfb672 2014-12-23T17:27:46.550+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:46-5499a6129a8f659f34bfb673", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355666550), what: "moveChunk.start", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:46.605+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 1|2||5499a60e00df4929841901d0, current metadata version is 1|2||5499a60e00df4929841901d0 2014-12-23T17:27:46.606+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-24 already up to date (shard version : 1|2||5499a60e00df4929841901d0, took 0ms) 2014-12-23T17:27:46.606+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a60e00df4929841901d0 2014-12-23T17:27:46.607+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:46.609+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:46.612+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:46.616+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:46.616+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:46.616+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:46.616+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a60e00df4929841901d0 2014-12-23T17:27:46.668+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:46.668+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a60e00df4929841901d0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-24' 2014-12-23T17:27:46.975+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:46-5499a6129a8f659f34bfb674", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355666975), what: "moveChunk.commit", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:47.046+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:47.046+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:47.046+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-24 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3301 2014-12-23T17:27:47.046+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-24 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:47.046+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:47.253+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:47.253+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:47-5499a6139a8f659f34bfb675", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355667253), what: "moveChunk.from", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 662, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 429, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:47.306+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60e00df4929841901d0') } ntoreturn:1 keyUpdates:0 reslen:37 1362ms 2014-12-23T17:27:47.307+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:47.308+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60e00df4929841901d0') } 2014-12-23T17:27:47.630+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6139a8f659f34bfb676 2014-12-23T17:27:47.630+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:47-5499a6139a8f659f34bfb677", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355667630), what: "moveChunk.start", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:47.680+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 2|0||5499a60e00df4929841901d0, current metadata version is 2|0||5499a60e00df4929841901d0 2014-12-23T17:27:47.681+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-24 from shard version 2|0||5499a60e00df4929841901d0 to shard version 2|1||5499a60e00df4929841901d0 2014-12-23T17:27:47.681+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a60e00df4929841901d0, took 0ms 2014-12-23T17:27:47.681+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a60e00df4929841901d0 2014-12-23T17:27:47.681+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:47.683+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:47.686+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:47.690+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:47.699+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:47.715+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:47.715+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:47.715+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:47.715+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a60e00df4929841901d0 2014-12-23T17:27:47.764+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:47.764+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a60e00df4929841901d0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-24' 2014-12-23T17:27:47.958+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:47-5499a6139a8f659f34bfb678", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355667958), what: "moveChunk.commit", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:48.015+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:48.015+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:48.015+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-24 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3311 2014-12-23T17:27:48.015+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-24 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:48.015+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:48.171+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:48.171+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:48-5499a6149a8f659f34bfb679", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355668171), what: "moveChunk.from", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 373, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 299, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:48.228+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a60e00df4929841901d0') } ntoreturn:1 keyUpdates:0 reslen:37 920ms 2014-12-23T17:27:48.229+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-24", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-24-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a60e00df4929841901d0') } 2014-12-23T17:27:48.763+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6149a8f659f34bfb67a 2014-12-23T17:27:48.763+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 3|0||5499a60e00df4929841901d0, current metadata version is 3|0||5499a60e00df4929841901d0 2014-12-23T17:27:48.764+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-24 from shard version 3|0||5499a60e00df4929841901d0 to shard version 3|1||5499a60e00df4929841901d0 2014-12-23T17:27:48.764+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a60e00df4929841901d0, took 0ms 2014-12-23T17:27:48.764+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a60e00df4929841901d0 2014-12-23T17:27:49.068+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:49-5499a6159a8f659f34bfb67b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355669068), what: "split", ns: "benchdb1.COL-24", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a60e00df4929841901d0') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a60e00df4929841901d0') } } } 2014-12-23T17:27:49.495+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:49.495+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-24", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-24-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a60e00df4929841901d0') } ntoreturn:1 keyUpdates:0 reslen:37 1265ms 2014-12-23T17:27:52.083+0000 I INDEX [conn2] build index on: benchdb1.COL-25 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-25" } 2014-12-23T17:27:52.083+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:52.084+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:52.381+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-25 with requested shard version 1|2||5499a61500df4929841901d5, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:52.382+0000 I SHARDING [conn2] collection benchdb1.COL-25 was previously unsharded, new metadata loaded with shard version 1|2||5499a61500df4929841901d5 2014-12-23T17:27:52.382+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a61500df4929841901d5, took 0ms 2014-12-23T17:27:52.418+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:52.418+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61500df4929841901d5') } 2014-12-23T17:27:53.025+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6189a8f659f34bfb67d 2014-12-23T17:27:53.025+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:53-5499a6199a8f659f34bfb67e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355673025), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:53.090+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 1|2||5499a61500df4929841901d5, current metadata version is 1|2||5499a61500df4929841901d5 2014-12-23T17:27:53.091+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-25 already up to date (shard version : 1|2||5499a61500df4929841901d5, took 0ms) 2014-12-23T17:27:53.091+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a61500df4929841901d5 2014-12-23T17:27:53.091+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:53.094+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:53.096+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:53.101+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:53.101+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:53.101+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:53.101+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a61500df4929841901d5 2014-12-23T17:27:53.158+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:53.158+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a61500df4929841901d5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-25' 2014-12-23T17:27:53.423+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:53-5499a6199a8f659f34bfb67f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355673423), what: "moveChunk.commit", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:53.491+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:53.491+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:53.492+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-25 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3329 2014-12-23T17:27:53.492+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-25 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:27:53.492+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:53.794+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:53.795+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:53-5499a6199a8f659f34bfb680", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355673795), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 390, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:27:53.853+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61500df4929841901d5') } ntoreturn:1 keyUpdates:0 reslen:37 1434ms 2014-12-23T17:27:53.854+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:53.855+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61500df4929841901d5') } 2014-12-23T17:27:54.461+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6199a8f659f34bfb681 2014-12-23T17:27:54.461+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:54-5499a61a9a8f659f34bfb682", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355674461), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:27:54.523+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 2|0||5499a61500df4929841901d5, current metadata version is 2|0||5499a61500df4929841901d5 2014-12-23T17:27:54.524+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-25 from shard version 2|0||5499a61500df4929841901d5 to shard version 2|1||5499a61500df4929841901d5 2014-12-23T17:27:54.524+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a61500df4929841901d5, took 0ms 2014-12-23T17:27:54.524+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a61500df4929841901d5 2014-12-23T17:27:54.524+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:54.527+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:54.529+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:54.533+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:54.542+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:54.558+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:54.558+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:54.559+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:54.559+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a61500df4929841901d5 2014-12-23T17:27:54.610+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:54.610+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a61500df4929841901d5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-25' 2014-12-23T17:27:54.914+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:54-5499a61a9a8f659f34bfb683", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355674914), what: "moveChunk.commit", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:27:54.975+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:54.975+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:27:54.975+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-25 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3337 2014-12-23T17:27:54.975+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-25 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:27:54.975+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:27:55.278+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:55.278+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:55-5499a61b9a8f659f34bfb684", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355675278), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 416, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:27:55.336+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61500df4929841901d5') } ntoreturn:1 keyUpdates:0 reslen:37 1481ms 2014-12-23T17:27:55.338+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a61500df4929841901d5') } 2014-12-23T17:27:55.944+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a61b9a8f659f34bfb685 2014-12-23T17:27:55.944+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 3|0||5499a61500df4929841901d5, current metadata version is 3|0||5499a61500df4929841901d5 2014-12-23T17:27:55.945+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-25 from shard version 3|0||5499a61500df4929841901d5 to shard version 3|1||5499a61500df4929841901d5 2014-12-23T17:27:55.945+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a61500df4929841901d5, took 0ms 2014-12-23T17:27:55.945+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a61500df4929841901d5 2014-12-23T17:27:56.250+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:56-5499a61c9a8f659f34bfb686", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355676250), what: "split", ns: "benchdb1.COL-25", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a61500df4929841901d5') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a61500df4929841901d5') } } } 2014-12-23T17:27:56.620+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:27:56.620+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a61500df4929841901d5') } ntoreturn:1 keyUpdates:0 reslen:37 1282ms 2014-12-23T17:27:59.111+0000 I INDEX [conn2] build index on: benchdb1.COL-26 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-26" } 2014-12-23T17:27:59.111+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:27:59.112+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:27:59.393+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-26 with requested shard version 1|2||5499a61c00df4929841901d9, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:27:59.394+0000 I SHARDING [conn2] collection benchdb1.COL-26 was previously unsharded, new metadata loaded with shard version 1|2||5499a61c00df4929841901d9 2014-12-23T17:27:59.394+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a61c00df4929841901d9, took 0ms 2014-12-23T17:27:59.435+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:27:59.435+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61c00df4929841901d9') } 2014-12-23T17:27:59.658+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a61f9a8f659f34bfb688 2014-12-23T17:27:59.658+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:59-5499a61f9a8f659f34bfb689", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355679658), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:27:59.712+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 1|2||5499a61c00df4929841901d9, current metadata version is 1|2||5499a61c00df4929841901d9 2014-12-23T17:27:59.713+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-26 already up to date (shard version : 1|2||5499a61c00df4929841901d9, took 0ms) 2014-12-23T17:27:59.713+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a61c00df4929841901d9 2014-12-23T17:27:59.713+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:27:59.716+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:59.718+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:59.723+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:27:59.723+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:27:59.723+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:27:59.723+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a61c00df4929841901d9 2014-12-23T17:27:59.771+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:27:59.772+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a61c00df4929841901d9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-26' 2014-12-23T17:27:59.976+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:27:59-5499a61f9a8f659f34bfb68a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355679976), what: "moveChunk.commit", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:00.062+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:00.062+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:00.063+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-26 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3358 2014-12-23T17:28:00.063+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-26 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:00.063+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:00.229+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:00.229+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:00-5499a6209a8f659f34bfb68b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355680229), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 277, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 339, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:00.283+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61c00df4929841901d9') } ntoreturn:1 keyUpdates:0 reslen:37 847ms 2014-12-23T17:28:00.284+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:00.285+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61c00df4929841901d9') } 2014-12-23T17:28:00.680+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6209a8f659f34bfb68c 2014-12-23T17:28:00.680+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:00-5499a6209a8f659f34bfb68d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355680680), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:00.737+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 2|0||5499a61c00df4929841901d9, current metadata version is 2|0||5499a61c00df4929841901d9 2014-12-23T17:28:00.737+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-26 from shard version 2|0||5499a61c00df4929841901d9 to shard version 2|1||5499a61c00df4929841901d9 2014-12-23T17:28:00.738+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a61c00df4929841901d9, took 0ms 2014-12-23T17:28:00.738+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a61c00df4929841901d9 2014-12-23T17:28:00.738+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:00.740+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:00.742+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:00.747+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:00.755+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:00.772+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:00.772+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:00.772+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:00.772+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a61c00df4929841901d9 2014-12-23T17:28:00.832+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:00.832+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a61c00df4929841901d9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-26' 2014-12-23T17:28:01.136+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:01-5499a6219a8f659f34bfb68e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355681136), what: "moveChunk.commit", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:01.197+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:01.197+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:01.197+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-26 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3366 2014-12-23T17:28:01.197+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-26 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:01.197+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:01.500+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:01.500+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:01-5499a6219a8f659f34bfb68f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355681500), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 452, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 425, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:01.558+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a61c00df4929841901d9') } ntoreturn:1 keyUpdates:0 reslen:37 1273ms 2014-12-23T17:28:01.559+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a61c00df4929841901d9') } 2014-12-23T17:28:02.166+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6219a8f659f34bfb690 2014-12-23T17:28:02.166+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 3|0||5499a61c00df4929841901d9, current metadata version is 3|0||5499a61c00df4929841901d9 2014-12-23T17:28:02.167+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-26 from shard version 3|0||5499a61c00df4929841901d9 to shard version 3|1||5499a61c00df4929841901d9 2014-12-23T17:28:02.167+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a61c00df4929841901d9, took 0ms 2014-12-23T17:28:02.167+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a61c00df4929841901d9 2014-12-23T17:28:02.472+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:02-5499a6229a8f659f34bfb691", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355682472), what: "split", ns: "benchdb1.COL-26", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a61c00df4929841901d9') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a61c00df4929841901d9') } } } 2014-12-23T17:28:02.851+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:02.851+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a61c00df4929841901d9') } ntoreturn:1 keyUpdates:0 reslen:37 1291ms 2014-12-23T17:28:05.421+0000 I INDEX [conn2] build index on: benchdb1.COL-27 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-27" } 2014-12-23T17:28:05.421+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:05.422+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:05.742+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-27 with requested shard version 1|2||5499a62200df4929841901dd, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:05.744+0000 I SHARDING [conn2] collection benchdb1.COL-27 was previously unsharded, new metadata loaded with shard version 1|2||5499a62200df4929841901dd 2014-12-23T17:28:05.744+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a62200df4929841901dd, took 1ms 2014-12-23T17:28:05.789+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:05.790+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62200df4929841901dd') } 2014-12-23T17:28:06.398+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6259a8f659f34bfb693 2014-12-23T17:28:06.398+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:06-5499a6269a8f659f34bfb694", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355686398), what: "moveChunk.start", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:06.464+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 1|2||5499a62200df4929841901dd, current metadata version is 1|2||5499a62200df4929841901dd 2014-12-23T17:28:06.465+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-27 already up to date (shard version : 1|2||5499a62200df4929841901dd, took 1ms) 2014-12-23T17:28:06.465+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a62200df4929841901dd 2014-12-23T17:28:06.465+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:06.468+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:06.470+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:06.475+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:06.475+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:06.475+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:06.475+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a62200df4929841901dd 2014-12-23T17:28:06.536+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:06.536+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a62200df4929841901dd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-27' 2014-12-23T17:28:06.841+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:06-5499a6269a8f659f34bfb695", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355686841), what: "moveChunk.commit", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:06.892+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:06.892+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:06.892+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-27 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3385 2014-12-23T17:28:06.892+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-27 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:06.892+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:07.196+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:07.196+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:07-5499a6279a8f659f34bfb696", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355687196), what: "moveChunk.from", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 674, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 417, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:07.251+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62200df4929841901dd') } ntoreturn:1 keyUpdates:0 reslen:37 1461ms 2014-12-23T17:28:07.253+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:07.253+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62200df4929841901dd') } 2014-12-23T17:28:07.811+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6279a8f659f34bfb697 2014-12-23T17:28:07.811+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:07-5499a6279a8f659f34bfb698", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355687811), what: "moveChunk.start", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:07.882+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 2|0||5499a62200df4929841901dd, current metadata version is 2|0||5499a62200df4929841901dd 2014-12-23T17:28:07.882+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:28:07 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:28:07.883+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-27 from shard version 2|0||5499a62200df4929841901dd to shard version 2|1||5499a62200df4929841901dd 2014-12-23T17:28:07.883+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a62200df4929841901dd, took 0ms 2014-12-23T17:28:07.883+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a62200df4929841901dd 2014-12-23T17:28:07.883+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:07.886+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:07.888+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:07.893+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:07.901+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:07.918+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:07.918+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:07.918+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:07.918+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a62200df4929841901dd 2014-12-23T17:28:07.981+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:07.981+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a62200df4929841901dd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-27' 2014-12-23T17:28:08.286+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:08-5499a6289a8f659f34bfb699", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355688286), what: "moveChunk.commit", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:08.342+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:08.342+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:08.343+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-27 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3393 2014-12-23T17:28:08.343+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-27 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:08.343+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:08.646+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:08.646+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:08-5499a6289a8f659f34bfb69a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355688646), what: "moveChunk.from", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 630, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:08.700+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62200df4929841901dd') } ntoreturn:1 keyUpdates:0 reslen:37 1447ms 2014-12-23T17:28:08.702+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-27", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-27-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a62200df4929841901dd') } 2014-12-23T17:28:09.311+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6289a8f659f34bfb69b 2014-12-23T17:28:09.311+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 3|0||5499a62200df4929841901dd, current metadata version is 3|0||5499a62200df4929841901dd 2014-12-23T17:28:09.312+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-27 from shard version 3|0||5499a62200df4929841901dd to shard version 3|1||5499a62200df4929841901dd 2014-12-23T17:28:09.312+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a62200df4929841901dd, took 0ms 2014-12-23T17:28:09.312+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a62200df4929841901dd 2014-12-23T17:28:09.595+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:09-5499a6299a8f659f34bfb69c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355689595), what: "split", ns: "benchdb1.COL-27", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a62200df4929841901dd') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a62200df4929841901dd') } } } 2014-12-23T17:28:09.976+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:09.976+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-27", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-27-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a62200df4929841901dd') } ntoreturn:1 keyUpdates:0 reslen:37 1273ms 2014-12-23T17:28:11.961+0000 I INDEX [conn2] build index on: benchdb1.COL-28 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-28" } 2014-12-23T17:28:11.961+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:11.962+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:12.265+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-28 with requested shard version 1|2||5499a62800df4929841901e1, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:12.266+0000 I SHARDING [conn2] collection benchdb1.COL-28 was previously unsharded, new metadata loaded with shard version 1|2||5499a62800df4929841901e1 2014-12-23T17:28:12.266+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a62800df4929841901e1, took 0ms 2014-12-23T17:28:12.314+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:12.315+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62800df4929841901e1') } 2014-12-23T17:28:12.619+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a62c9a8f659f34bfb69e 2014-12-23T17:28:12.619+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:12-5499a62c9a8f659f34bfb69f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355692619), what: "moveChunk.start", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:12.669+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 1|2||5499a62800df4929841901e1, current metadata version is 1|2||5499a62800df4929841901e1 2014-12-23T17:28:12.670+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-28 already up to date (shard version : 1|2||5499a62800df4929841901e1, took 0ms) 2014-12-23T17:28:12.670+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a62800df4929841901e1 2014-12-23T17:28:12.670+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:12.673+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:12.675+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:12.680+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:12.680+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:12.680+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:12.680+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a62800df4929841901e1 2014-12-23T17:28:12.753+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:12.753+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a62800df4929841901e1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-28' 2014-12-23T17:28:13.014+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:13-5499a62d9a8f659f34bfb6a0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355693014), what: "moveChunk.commit", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:13.057+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:13.057+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:13.058+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-28 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3414 2014-12-23T17:28:13.058+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-28 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:13.058+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:13.262+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:13.262+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:13-5499a62d9a8f659f34bfb6a1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355693262), what: "moveChunk.from", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 354, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 377, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:13.330+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62800df4929841901e1') } ntoreturn:1 keyUpdates:0 reslen:37 1015ms 2014-12-23T17:28:13.331+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:13.331+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62800df4929841901e1') } 2014-12-23T17:28:13.881+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a62d9a8f659f34bfb6a2 2014-12-23T17:28:13.881+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:13-5499a62d9a8f659f34bfb6a3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355693881), what: "moveChunk.start", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:13.946+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 2|0||5499a62800df4929841901e1, current metadata version is 2|0||5499a62800df4929841901e1 2014-12-23T17:28:13.947+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-28 from shard version 2|0||5499a62800df4929841901e1 to shard version 2|1||5499a62800df4929841901e1 2014-12-23T17:28:13.947+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a62800df4929841901e1, took 0ms 2014-12-23T17:28:13.947+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a62800df4929841901e1 2014-12-23T17:28:13.947+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:13.949+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:13.952+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:13.956+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:13.964+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:13.981+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:13.981+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:13.981+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:13.981+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a62800df4929841901e1 2014-12-23T17:28:14.042+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:14.042+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a62800df4929841901e1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-28' 2014-12-23T17:28:14.347+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:14-5499a62e9a8f659f34bfb6a4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355694347), what: "moveChunk.commit", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:14.406+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:14.406+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:14.406+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-28 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3422 2014-12-23T17:28:14.406+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-28 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:14.406+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:14.709+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:14.709+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:14-5499a62e9a8f659f34bfb6a5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355694709), what: "moveChunk.from", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 615, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:14.772+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62800df4929841901e1') } ntoreturn:1 keyUpdates:0 reslen:37 1440ms 2014-12-23T17:28:14.774+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-28", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-28-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a62800df4929841901e1') } 2014-12-23T17:28:15.330+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a62e9a8f659f34bfb6a6 2014-12-23T17:28:15.330+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 3|0||5499a62800df4929841901e1, current metadata version is 3|0||5499a62800df4929841901e1 2014-12-23T17:28:15.330+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-28 from shard version 3|0||5499a62800df4929841901e1 to shard version 3|1||5499a62800df4929841901e1 2014-12-23T17:28:15.331+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a62800df4929841901e1, took 0ms 2014-12-23T17:28:15.331+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a62800df4929841901e1 2014-12-23T17:28:15.635+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:15-5499a62f9a8f659f34bfb6a7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355695635), what: "split", ns: "benchdb1.COL-28", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a62800df4929841901e1') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a62800df4929841901e1') } } } 2014-12-23T17:28:16.019+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:16.019+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-28", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-28-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a62800df4929841901e1') } ntoreturn:1 keyUpdates:0 reslen:37 1245ms 2014-12-23T17:28:18.600+0000 I INDEX [conn2] build index on: benchdb1.COL-29 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-29" } 2014-12-23T17:28:18.600+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:18.601+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:18.922+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-29 with requested shard version 1|2||5499a62f00df4929841901e5, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:18.923+0000 I SHARDING [conn2] collection benchdb1.COL-29 was previously unsharded, new metadata loaded with shard version 1|2||5499a62f00df4929841901e5 2014-12-23T17:28:18.923+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a62f00df4929841901e5, took 0ms 2014-12-23T17:28:18.967+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:18.967+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62f00df4929841901e5') } 2014-12-23T17:28:19.525+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6329a8f659f34bfb6a9 2014-12-23T17:28:19.526+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:19-5499a6339a8f659f34bfb6aa", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355699526), what: "moveChunk.start", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:19.585+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 1|2||5499a62f00df4929841901e5, current metadata version is 1|2||5499a62f00df4929841901e5 2014-12-23T17:28:19.585+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-29 already up to date (shard version : 1|2||5499a62f00df4929841901e5, took 0ms) 2014-12-23T17:28:19.585+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a62f00df4929841901e5 2014-12-23T17:28:19.586+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:19.589+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:19.591+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:19.595+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:19.595+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:19.596+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:19.596+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a62f00df4929841901e5 2014-12-23T17:28:19.654+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:19.654+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a62f00df4929841901e5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-29' 2014-12-23T17:28:19.958+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:19-5499a6339a8f659f34bfb6ab", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355699958), what: "moveChunk.commit", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:20.011+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:20.011+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:20.011+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-29 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3441 2014-12-23T17:28:20.011+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-29 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:20.011+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:20.314+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:20.314+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:20-5499a6349a8f659f34bfb6ac", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355700314), what: "moveChunk.from", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 618, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 415, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:20.373+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62f00df4929841901e5') } ntoreturn:1 keyUpdates:0 reslen:37 1406ms 2014-12-23T17:28:20.375+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:20.375+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62f00df4929841901e5') } 2014-12-23T17:28:20.981+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6349a8f659f34bfb6ad 2014-12-23T17:28:20.981+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:20-5499a6349a8f659f34bfb6ae", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355700981), what: "moveChunk.start", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:21.036+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 2|0||5499a62f00df4929841901e5, current metadata version is 2|0||5499a62f00df4929841901e5 2014-12-23T17:28:21.037+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-29 from shard version 2|0||5499a62f00df4929841901e5 to shard version 2|1||5499a62f00df4929841901e5 2014-12-23T17:28:21.037+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a62f00df4929841901e5, took 0ms 2014-12-23T17:28:21.037+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a62f00df4929841901e5 2014-12-23T17:28:21.037+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:21.039+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:21.042+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:21.046+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:21.054+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:21.071+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:21.071+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:21.071+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:21.071+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a62f00df4929841901e5 2014-12-23T17:28:21.123+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:21.123+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a62f00df4929841901e5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-29' 2014-12-23T17:28:21.428+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:21-5499a6359a8f659f34bfb6af", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355701428), what: "moveChunk.commit", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:21.482+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:21.482+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:21.482+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-29 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3449 2014-12-23T17:28:21.482+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-29 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:21.482+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:21.797+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:21.798+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:21-5499a6359a8f659f34bfb6b0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355701797), what: "moveChunk.from", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 661, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 410, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:21.860+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a62f00df4929841901e5') } ntoreturn:1 keyUpdates:0 reslen:37 1485ms 2014-12-23T17:28:21.863+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-29", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-29-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a62f00df4929841901e5') } 2014-12-23T17:28:22.469+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6359a8f659f34bfb6b1 2014-12-23T17:28:22.469+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 3|0||5499a62f00df4929841901e5, current metadata version is 3|0||5499a62f00df4929841901e5 2014-12-23T17:28:22.470+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-29 from shard version 3|0||5499a62f00df4929841901e5 to shard version 3|1||5499a62f00df4929841901e5 2014-12-23T17:28:22.470+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a62f00df4929841901e5, took 0ms 2014-12-23T17:28:22.470+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a62f00df4929841901e5 2014-12-23T17:28:22.775+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:22-5499a6369a8f659f34bfb6b2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355702774), what: "split", ns: "benchdb1.COL-29", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a62f00df4929841901e5') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a62f00df4929841901e5') } } } 2014-12-23T17:28:23.156+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:23.156+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-29", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-29-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a62f00df4929841901e5') } ntoreturn:1 keyUpdates:0 reslen:37 1293ms 2014-12-23T17:28:24.836+0000 I INDEX [conn2] build index on: benchdb1.COL-30 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-30" } 2014-12-23T17:28:24.836+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:24.837+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:25.102+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-30 with requested shard version 1|2||5499a63500df4929841901e9, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:25.103+0000 I SHARDING [conn2] collection benchdb1.COL-30 was previously unsharded, new metadata loaded with shard version 1|2||5499a63500df4929841901e9 2014-12-23T17:28:25.103+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a63500df4929841901e9, took 0ms 2014-12-23T17:28:25.145+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:25.145+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63500df4929841901e9') } 2014-12-23T17:28:25.519+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6399a8f659f34bfb6b4 2014-12-23T17:28:25.519+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:25-5499a6399a8f659f34bfb6b5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355705519), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:25.567+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|2||5499a63500df4929841901e9, current metadata version is 1|2||5499a63500df4929841901e9 2014-12-23T17:28:25.568+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|2||5499a63500df4929841901e9, took 0ms) 2014-12-23T17:28:25.568+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a63500df4929841901e9 2014-12-23T17:28:25.568+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:25.571+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:25.573+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:25.578+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:25.578+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:25.578+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:25.578+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a63500df4929841901e9 2014-12-23T17:28:25.633+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:25.633+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a63500df4929841901e9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-30' 2014-12-23T17:28:25.938+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:25-5499a6399a8f659f34bfb6b6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355705938), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:25.992+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:25.992+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:25.992+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-30 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3470 2014-12-23T17:28:25.992+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:25.992+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:26.295+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:26.295+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:26-5499a63a9a8f659f34bfb6b7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355706295), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 422, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:26.367+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63500df4929841901e9') } ntoreturn:1 keyUpdates:0 reslen:37 1221ms 2014-12-23T17:28:26.368+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:26.369+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63500df4929841901e9') } 2014-12-23T17:28:26.975+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a63a9a8f659f34bfb6b8 2014-12-23T17:28:26.975+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:26-5499a63a9a8f659f34bfb6b9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355706975), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:27.040+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 2|0||5499a63500df4929841901e9, current metadata version is 2|0||5499a63500df4929841901e9 2014-12-23T17:28:27.041+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-30 from shard version 2|0||5499a63500df4929841901e9 to shard version 2|1||5499a63500df4929841901e9 2014-12-23T17:28:27.041+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a63500df4929841901e9, took 0ms 2014-12-23T17:28:27.041+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a63500df4929841901e9 2014-12-23T17:28:27.041+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:27.043+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:27.046+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:27.050+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:27.059+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:27.075+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:27.075+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:27.075+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:27.075+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a63500df4929841901e9 2014-12-23T17:28:27.132+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:27.132+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a63500df4929841901e9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-30' 2014-12-23T17:28:27.437+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:27-5499a63b9a8f659f34bfb6ba", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355707437), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:27.493+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:27.493+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:27.493+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-30 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3478 2014-12-23T17:28:27.493+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:27.493+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:27.796+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:27.796+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:27-5499a63b9a8f659f34bfb6bb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355707796), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 417, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:27.857+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63500df4929841901e9') } ntoreturn:1 keyUpdates:0 reslen:37 1488ms 2014-12-23T17:28:27.859+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a63500df4929841901e9') } 2014-12-23T17:28:28.465+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a63b9a8f659f34bfb6bc 2014-12-23T17:28:28.465+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 3|0||5499a63500df4929841901e9, current metadata version is 3|0||5499a63500df4929841901e9 2014-12-23T17:28:28.466+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-30 from shard version 3|0||5499a63500df4929841901e9 to shard version 3|1||5499a63500df4929841901e9 2014-12-23T17:28:28.466+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a63500df4929841901e9, took 0ms 2014-12-23T17:28:28.466+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a63500df4929841901e9 2014-12-23T17:28:28.770+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:28-5499a63c9a8f659f34bfb6bd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355708770), what: "split", ns: "benchdb1.COL-30", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a63500df4929841901e9') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a63500df4929841901e9') } } } 2014-12-23T17:28:29.144+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:29.144+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a63500df4929841901e9') } ntoreturn:1 keyUpdates:0 reslen:37 1284ms 2014-12-23T17:28:31.680+0000 I INDEX [conn2] build index on: benchdb1.COL-31 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-31" } 2014-12-23T17:28:31.680+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:31.681+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:31.968+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-31 with requested shard version 1|2||5499a63c00df4929841901ed, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:31.969+0000 I SHARDING [conn2] collection benchdb1.COL-31 was previously unsharded, new metadata loaded with shard version 1|2||5499a63c00df4929841901ed 2014-12-23T17:28:31.969+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a63c00df4929841901ed, took 0ms 2014-12-23T17:28:32.009+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:32.010+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63c00df4929841901ed') } 2014-12-23T17:28:32.618+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6409a8f659f34bfb6bf 2014-12-23T17:28:32.618+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:32-5499a6409a8f659f34bfb6c0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355712618), what: "moveChunk.start", ns: "benchdb1.COL-31", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:32.681+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-31 based on current shard version 1|2||5499a63c00df4929841901ed, current metadata version is 1|2||5499a63c00df4929841901ed 2014-12-23T17:28:32.682+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-31 already up to date (shard version : 1|2||5499a63c00df4929841901ed, took 0ms) 2014-12-23T17:28:32.682+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a63c00df4929841901ed 2014-12-23T17:28:32.683+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:32.685+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:32.688+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:32.692+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:32.692+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:32.692+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:32.692+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a63c00df4929841901ed 2014-12-23T17:28:32.751+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:32.751+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a63c00df4929841901ed through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-31' 2014-12-23T17:28:33.056+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:33-5499a6419a8f659f34bfb6c1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355713056), what: "moveChunk.commit", ns: "benchdb1.COL-31", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:33.110+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:33.110+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:33.111+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-31 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3497 2014-12-23T17:28:33.111+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-31 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:33.111+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:33.414+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:33.414+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:33-5499a6419a8f659f34bfb6c2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355713414), what: "moveChunk.from", ns: "benchdb1.COL-31", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 418, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:33.470+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63c00df4929841901ed') } ntoreturn:1 keyUpdates:0 reslen:37 1460ms 2014-12-23T17:28:33.471+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:33.472+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63c00df4929841901ed') } 2014-12-23T17:28:34.078+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6419a8f659f34bfb6c3 2014-12-23T17:28:34.079+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:34-5499a6429a8f659f34bfb6c4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355714079), what: "moveChunk.start", ns: "benchdb1.COL-31", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:34.143+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-31 based on current shard version 2|0||5499a63c00df4929841901ed, current metadata version is 2|0||5499a63c00df4929841901ed 2014-12-23T17:28:34.144+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-31 from shard version 2|0||5499a63c00df4929841901ed to shard version 2|1||5499a63c00df4929841901ed 2014-12-23T17:28:34.144+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a63c00df4929841901ed, took 0ms 2014-12-23T17:28:34.144+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a63c00df4929841901ed 2014-12-23T17:28:34.144+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:34.146+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:34.149+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:34.153+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:34.162+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:34.178+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:34.178+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:34.178+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:34.178+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a63c00df4929841901ed 2014-12-23T17:28:34.233+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:34.233+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a63c00df4929841901ed through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-31' 2014-12-23T17:28:34.538+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:34-5499a6429a8f659f34bfb6c5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355714538), what: "moveChunk.commit", ns: "benchdb1.COL-31", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:34.593+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:34.593+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:34.593+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-31 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3505 2014-12-23T17:28:34.593+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-31 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:34.593+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:34.896+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:34.896+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:34-5499a6429a8f659f34bfb6c6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355714896), what: "moveChunk.from", ns: "benchdb1.COL-31", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 414, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:34.950+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-31", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-31-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a63c00df4929841901ed') } ntoreturn:1 keyUpdates:0 reslen:37 1478ms 2014-12-23T17:28:34.951+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-31", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-31-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a63c00df4929841901ed') } 2014-12-23T17:28:35.558+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6429a8f659f34bfb6c7 2014-12-23T17:28:35.558+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-31 based on current shard version 3|0||5499a63c00df4929841901ed, current metadata version is 3|0||5499a63c00df4929841901ed 2014-12-23T17:28:35.559+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-31 from shard version 3|0||5499a63c00df4929841901ed to shard version 3|1||5499a63c00df4929841901ed 2014-12-23T17:28:35.559+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a63c00df4929841901ed, took 0ms 2014-12-23T17:28:35.559+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a63c00df4929841901ed 2014-12-23T17:28:35.831+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:35-5499a6439a8f659f34bfb6c8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355715831), what: "split", ns: "benchdb1.COL-31", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a63c00df4929841901ed') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a63c00df4929841901ed') } } } 2014-12-23T17:28:36.124+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-31/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:36.124+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-31", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-31-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a63c00df4929841901ed') } ntoreturn:1 keyUpdates:0 reslen:37 1172ms 2014-12-23T17:28:37.601+0000 I INDEX [conn2] build index on: benchdb1.COL-32 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-32" } 2014-12-23T17:28:37.601+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:37.602+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:37.905+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-32 with requested shard version 1|2||5499a64200df4929841901f2, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:37.906+0000 I SHARDING [conn2] collection benchdb1.COL-32 was previously unsharded, new metadata loaded with shard version 1|2||5499a64200df4929841901f2 2014-12-23T17:28:37.906+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a64200df4929841901f2, took 0ms 2014-12-23T17:28:37.948+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:37.949+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64200df4929841901f2') } 2014-12-23T17:28:38.355+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:28:37 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:28:38.506+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6459a8f659f34bfb6ca 2014-12-23T17:28:38.506+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:38-5499a6469a8f659f34bfb6cb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355718506), what: "moveChunk.start", ns: "benchdb1.COL-32", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:38.580+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-32 based on current shard version 1|2||5499a64200df4929841901f2, current metadata version is 1|2||5499a64200df4929841901f2 2014-12-23T17:28:38.581+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-32 already up to date (shard version : 1|2||5499a64200df4929841901f2, took 0ms) 2014-12-23T17:28:38.581+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a64200df4929841901f2 2014-12-23T17:28:38.582+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:38.584+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:38.587+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:38.591+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:38.591+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:38.592+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:38.592+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a64200df4929841901f2 2014-12-23T17:28:38.645+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:38.645+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a64200df4929841901f2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-32' 2014-12-23T17:28:38.950+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:38-5499a6469a8f659f34bfb6cc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355718950), what: "moveChunk.commit", ns: "benchdb1.COL-32", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:39.015+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:39.015+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:39.015+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-32 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3526 2014-12-23T17:28:39.015+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-32 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:39.015+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:39.318+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:39.318+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:39-5499a6479a8f659f34bfb6cd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355719318), what: "moveChunk.from", ns: "benchdb1.COL-32", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 632, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 423, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:39.372+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64200df4929841901f2') } ntoreturn:1 keyUpdates:0 reslen:37 1423ms 2014-12-23T17:28:39.373+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:39.374+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64200df4929841901f2') } 2014-12-23T17:28:39.934+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6479a8f659f34bfb6ce 2014-12-23T17:28:39.934+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:39-5499a6479a8f659f34bfb6cf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355719934), what: "moveChunk.start", ns: "benchdb1.COL-32", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:40.010+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-32 based on current shard version 2|0||5499a64200df4929841901f2, current metadata version is 2|0||5499a64200df4929841901f2 2014-12-23T17:28:40.011+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-32 from shard version 2|0||5499a64200df4929841901f2 to shard version 2|1||5499a64200df4929841901f2 2014-12-23T17:28:40.011+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a64200df4929841901f2, took 0ms 2014-12-23T17:28:40.011+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a64200df4929841901f2 2014-12-23T17:28:40.011+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:40.013+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:40.015+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:40.020+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:40.028+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:40.045+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:40.045+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:40.045+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:40.045+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a64200df4929841901f2 2014-12-23T17:28:40.101+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:40.101+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a64200df4929841901f2 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-32' 2014-12-23T17:28:40.405+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:40-5499a6489a8f659f34bfb6d0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355720405), what: "moveChunk.commit", ns: "benchdb1.COL-32", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:40.459+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:40.459+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:40.459+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-32 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3534 2014-12-23T17:28:40.459+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-32 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:40.459+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:40.727+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:40.727+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:40-5499a6489a8f659f34bfb6d1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355720727), what: "moveChunk.from", ns: "benchdb1.COL-32", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 636, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 414, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:40.780+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-32", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-32-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64200df4929841901f2') } ntoreturn:1 keyUpdates:0 reslen:37 1407ms 2014-12-23T17:28:40.782+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-32", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-32-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a64200df4929841901f2') } 2014-12-23T17:28:41.389+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6489a8f659f34bfb6d2 2014-12-23T17:28:41.389+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-32 based on current shard version 3|0||5499a64200df4929841901f2, current metadata version is 3|0||5499a64200df4929841901f2 2014-12-23T17:28:41.390+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-32 from shard version 3|0||5499a64200df4929841901f2 to shard version 3|1||5499a64200df4929841901f2 2014-12-23T17:28:41.390+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a64200df4929841901f2, took 0ms 2014-12-23T17:28:41.390+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a64200df4929841901f2 2014-12-23T17:28:41.694+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:41-5499a6499a8f659f34bfb6d3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355721694), what: "split", ns: "benchdb1.COL-32", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a64200df4929841901f2') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a64200df4929841901f2') } } } 2014-12-23T17:28:42.072+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-32/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:42.072+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-32", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-32-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a64200df4929841901f2') } ntoreturn:1 keyUpdates:0 reslen:37 1289ms 2014-12-23T17:28:44.625+0000 I INDEX [conn2] build index on: benchdb1.COL-33 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-33" } 2014-12-23T17:28:44.625+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:44.626+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:44.961+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-33 with requested shard version 1|2||5499a64900df4929841901f5, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:44.962+0000 I SHARDING [conn2] collection benchdb1.COL-33 was previously unsharded, new metadata loaded with shard version 1|2||5499a64900df4929841901f5 2014-12-23T17:28:44.962+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a64900df4929841901f5, took 0ms 2014-12-23T17:28:45.001+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:45.001+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64900df4929841901f5') } 2014-12-23T17:28:45.583+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a64c9a8f659f34bfb6d5 2014-12-23T17:28:45.583+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:45-5499a64d9a8f659f34bfb6d6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355725583), what: "moveChunk.start", ns: "benchdb1.COL-33", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:45.647+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-33 based on current shard version 1|2||5499a64900df4929841901f5, current metadata version is 1|2||5499a64900df4929841901f5 2014-12-23T17:28:45.648+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-33 already up to date (shard version : 1|2||5499a64900df4929841901f5, took 0ms) 2014-12-23T17:28:45.648+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a64900df4929841901f5 2014-12-23T17:28:45.649+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:45.651+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:45.654+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:45.658+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:45.658+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:45.658+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:45.658+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a64900df4929841901f5 2014-12-23T17:28:45.714+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:45.714+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a64900df4929841901f5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-33' 2014-12-23T17:28:46.018+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:46-5499a64e9a8f659f34bfb6d7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355726018), what: "moveChunk.commit", ns: "benchdb1.COL-33", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:46.085+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:46.085+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:46.085+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-33 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3553 2014-12-23T17:28:46.085+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-33 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:46.085+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:46.388+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:46.388+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:46-5499a64e9a8f659f34bfb6d8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355726388), what: "moveChunk.from", ns: "benchdb1.COL-33", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 647, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 427, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:46.448+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64900df4929841901f5') } ntoreturn:1 keyUpdates:0 reslen:37 1447ms 2014-12-23T17:28:46.449+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:46.450+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64900df4929841901f5') } 2014-12-23T17:28:47.056+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a64e9a8f659f34bfb6d9 2014-12-23T17:28:47.056+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:47-5499a64f9a8f659f34bfb6da", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355727056), what: "moveChunk.start", ns: "benchdb1.COL-33", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:47.117+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-33 based on current shard version 2|0||5499a64900df4929841901f5, current metadata version is 2|0||5499a64900df4929841901f5 2014-12-23T17:28:47.118+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-33 from shard version 2|0||5499a64900df4929841901f5 to shard version 2|1||5499a64900df4929841901f5 2014-12-23T17:28:47.118+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a64900df4929841901f5, took 0ms 2014-12-23T17:28:47.118+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a64900df4929841901f5 2014-12-23T17:28:47.118+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:47.121+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:47.123+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:47.128+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:47.136+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:47.152+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:47.153+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:47.153+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:47.153+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a64900df4929841901f5 2014-12-23T17:28:47.210+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:47.210+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a64900df4929841901f5 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-33' 2014-12-23T17:28:47.515+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:47-5499a64f9a8f659f34bfb6db", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355727515), what: "moveChunk.commit", ns: "benchdb1.COL-33", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:47.571+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:47.571+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:47.571+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-33 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3561 2014-12-23T17:28:47.571+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-33 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:47.571+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:47.827+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:47.827+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:47-5499a64f9a8f659f34bfb6dc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355727827), what: "moveChunk.from", ns: "benchdb1.COL-33", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 418, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:47.895+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-33", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-33-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64900df4929841901f5') } ntoreturn:1 keyUpdates:0 reslen:37 1445ms 2014-12-23T17:28:47.897+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-33", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-33-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a64900df4929841901f5') } 2014-12-23T17:28:48.250+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a64f9a8f659f34bfb6dd 2014-12-23T17:28:48.250+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-33 based on current shard version 3|0||5499a64900df4929841901f5, current metadata version is 3|0||5499a64900df4929841901f5 2014-12-23T17:28:48.251+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-33 from shard version 3|0||5499a64900df4929841901f5 to shard version 3|1||5499a64900df4929841901f5 2014-12-23T17:28:48.251+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a64900df4929841901f5, took 0ms 2014-12-23T17:28:48.251+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a64900df4929841901f5 2014-12-23T17:28:48.432+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:48-5499a6509a8f659f34bfb6de", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355728432), what: "split", ns: "benchdb1.COL-33", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a64900df4929841901f5') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a64900df4929841901f5') } } } 2014-12-23T17:28:48.653+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-33/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:48.653+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-33", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-33-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a64900df4929841901f5') } ntoreturn:1 keyUpdates:0 reslen:37 755ms 2014-12-23T17:28:50.721+0000 I INDEX [conn2] build index on: benchdb1.COL-34 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-34" } 2014-12-23T17:28:50.721+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:50.722+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:51.047+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-34 with requested shard version 1|2||5499a64f00df4929841901fa, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:51.048+0000 I SHARDING [conn2] collection benchdb1.COL-34 was previously unsharded, new metadata loaded with shard version 1|2||5499a64f00df4929841901fa 2014-12-23T17:28:51.048+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a64f00df4929841901fa, took 0ms 2014-12-23T17:28:51.090+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:51.090+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64f00df4929841901fa') } 2014-12-23T17:28:51.697+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6539a8f659f34bfb6e0 2014-12-23T17:28:51.697+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:51-5499a6539a8f659f34bfb6e1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355731697), what: "moveChunk.start", ns: "benchdb1.COL-34", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:51.761+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-34 based on current shard version 1|2||5499a64f00df4929841901fa, current metadata version is 1|2||5499a64f00df4929841901fa 2014-12-23T17:28:51.761+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-34 already up to date (shard version : 1|2||5499a64f00df4929841901fa, took 0ms) 2014-12-23T17:28:51.761+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a64f00df4929841901fa 2014-12-23T17:28:51.762+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:51.764+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:51.767+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:51.771+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:51.771+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:51.772+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:51.772+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a64f00df4929841901fa 2014-12-23T17:28:51.824+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:51.824+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a64f00df4929841901fa through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-34' 2014-12-23T17:28:52.129+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:52-5499a6549a8f659f34bfb6e2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355732129), what: "moveChunk.commit", ns: "benchdb1.COL-34", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:52.194+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:52.195+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:52.195+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-34 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3582 2014-12-23T17:28:52.195+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-34 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:52.195+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:52.498+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:52.498+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:52-5499a6549a8f659f34bfb6e3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355732498), what: "moveChunk.from", ns: "benchdb1.COL-34", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 423, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:52.556+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64f00df4929841901fa') } ntoreturn:1 keyUpdates:0 reslen:37 1466ms 2014-12-23T17:28:52.558+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:52.559+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64f00df4929841901fa') } 2014-12-23T17:28:53.165+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6549a8f659f34bfb6e4 2014-12-23T17:28:53.165+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:53-5499a6559a8f659f34bfb6e5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355733165), what: "moveChunk.start", ns: "benchdb1.COL-34", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:28:53.233+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-34 based on current shard version 2|0||5499a64f00df4929841901fa, current metadata version is 2|0||5499a64f00df4929841901fa 2014-12-23T17:28:53.234+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-34 from shard version 2|0||5499a64f00df4929841901fa to shard version 2|1||5499a64f00df4929841901fa 2014-12-23T17:28:53.234+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a64f00df4929841901fa, took 0ms 2014-12-23T17:28:53.234+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a64f00df4929841901fa 2014-12-23T17:28:53.234+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:53.236+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:53.239+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:53.243+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:53.252+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:53.268+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:53.268+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:53.268+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:53.268+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a64f00df4929841901fa 2014-12-23T17:28:53.320+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:53.320+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a64f00df4929841901fa through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-34' 2014-12-23T17:28:53.578+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:53-5499a6559a8f659f34bfb6e6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355733578), what: "moveChunk.commit", ns: "benchdb1.COL-34", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:53.643+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:53.643+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:53.643+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-34 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3590 2014-12-23T17:28:53.643+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-34 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:28:53.643+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:53.946+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:53.946+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:53-5499a6559a8f659f34bfb6e7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355733946), what: "moveChunk.from", ns: "benchdb1.COL-34", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 675, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 374, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:28:53.998+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-34", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-34-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a64f00df4929841901fa') } ntoreturn:1 keyUpdates:0 reslen:37 1439ms 2014-12-23T17:28:54.000+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-34", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-34-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a64f00df4929841901fa') } 2014-12-23T17:28:54.606+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6559a8f659f34bfb6e8 2014-12-23T17:28:54.606+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-34 based on current shard version 3|0||5499a64f00df4929841901fa, current metadata version is 3|0||5499a64f00df4929841901fa 2014-12-23T17:28:54.607+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-34 from shard version 3|0||5499a64f00df4929841901fa to shard version 3|1||5499a64f00df4929841901fa 2014-12-23T17:28:54.607+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a64f00df4929841901fa, took 0ms 2014-12-23T17:28:54.607+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a64f00df4929841901fa 2014-12-23T17:28:54.912+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:54-5499a6569a8f659f34bfb6e9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355734911), what: "split", ns: "benchdb1.COL-34", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a64f00df4929841901fa') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a64f00df4929841901fa') } } } 2014-12-23T17:28:55.294+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-34/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:55.294+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-34", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-34-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a64f00df4929841901fa') } ntoreturn:1 keyUpdates:0 reslen:37 1294ms 2014-12-23T17:28:57.891+0000 I INDEX [conn2] build index on: benchdb1.COL-35 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-35" } 2014-12-23T17:28:57.891+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:28:57.892+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:28:58.203+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-35 with requested shard version 1|2||5499a65600df4929841901fd, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:28:58.204+0000 I SHARDING [conn2] collection benchdb1.COL-35 was previously unsharded, new metadata loaded with shard version 1|2||5499a65600df4929841901fd 2014-12-23T17:28:58.204+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a65600df4929841901fd, took 0ms 2014-12-23T17:28:58.255+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:58.256+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65600df4929841901fd') } 2014-12-23T17:28:58.863+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a65a9a8f659f34bfb6eb 2014-12-23T17:28:58.863+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:58-5499a65a9a8f659f34bfb6ec", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355738863), what: "moveChunk.start", ns: "benchdb1.COL-35", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:28:58.926+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-35 based on current shard version 1|2||5499a65600df4929841901fd, current metadata version is 1|2||5499a65600df4929841901fd 2014-12-23T17:28:58.927+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-35 already up to date (shard version : 1|2||5499a65600df4929841901fd, took 0ms) 2014-12-23T17:28:58.927+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a65600df4929841901fd 2014-12-23T17:28:58.927+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:28:58.930+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:58.932+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:58.936+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:28:58.937+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:28:58.937+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:28:58.937+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a65600df4929841901fd 2014-12-23T17:28:58.993+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:28:58.993+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a65600df4929841901fd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-35' 2014-12-23T17:28:59.298+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:59-5499a65b9a8f659f34bfb6ed", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355739298), what: "moveChunk.commit", ns: "benchdb1.COL-35", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:28:59.359+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:59.359+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:28:59.359+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-35 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3609 2014-12-23T17:28:59.359+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-35 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:28:59.359+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:28:59.614+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:28:59.614+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:28:59-5499a65b9a8f659f34bfb6ee", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355739614), what: "moveChunk.from", ns: "benchdb1.COL-35", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 422, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:28:59.687+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65600df4929841901fd') } ntoreturn:1 keyUpdates:0 reslen:37 1432ms 2014-12-23T17:28:59.689+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:28:59.689+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65600df4929841901fd') } 2014-12-23T17:29:00.163+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a65b9a8f659f34bfb6ef 2014-12-23T17:29:00.163+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:00-5499a65c9a8f659f34bfb6f0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355740163), what: "moveChunk.start", ns: "benchdb1.COL-35", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:00.227+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-35 based on current shard version 2|0||5499a65600df4929841901fd, current metadata version is 2|0||5499a65600df4929841901fd 2014-12-23T17:29:00.227+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-35 from shard version 2|0||5499a65600df4929841901fd to shard version 2|1||5499a65600df4929841901fd 2014-12-23T17:29:00.227+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a65600df4929841901fd, took 0ms 2014-12-23T17:29:00.227+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a65600df4929841901fd 2014-12-23T17:29:00.228+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:29:00.230+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:00.232+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:00.237+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:00.245+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:00.261+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:00.262+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:29:00.262+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:29:00.262+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a65600df4929841901fd 2014-12-23T17:29:00.315+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:00.315+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a65600df4929841901fd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-35' 2014-12-23T17:29:00.518+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:00-5499a65c9a8f659f34bfb6f1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355740518), what: "moveChunk.commit", ns: "benchdb1.COL-35", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:00.571+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:00.571+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:29:00.571+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-35 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3618 2014-12-23T17:29:00.571+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-35 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:00.571+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:00.712+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:00.712+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:00-5499a65c9a8f659f34bfb6f2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355740712), what: "moveChunk.from", ns: "benchdb1.COL-35", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 538, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 309, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:00.759+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-35", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-35-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65600df4929841901fd') } ntoreturn:1 keyUpdates:0 reslen:37 1069ms 2014-12-23T17:29:00.761+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-35", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-35-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a65600df4929841901fd') } 2014-12-23T17:29:01.110+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a65c9a8f659f34bfb6f3 2014-12-23T17:29:01.110+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-35 based on current shard version 3|0||5499a65600df4929841901fd, current metadata version is 3|0||5499a65600df4929841901fd 2014-12-23T17:29:01.111+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-35 from shard version 3|0||5499a65600df4929841901fd to shard version 3|1||5499a65600df4929841901fd 2014-12-23T17:29:01.111+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a65600df4929841901fd, took 0ms 2014-12-23T17:29:01.111+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a65600df4929841901fd 2014-12-23T17:29:01.288+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:01-5499a65d9a8f659f34bfb6f4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355741288), what: "split", ns: "benchdb1.COL-35", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a65600df4929841901fd') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a65600df4929841901fd') } } } 2014-12-23T17:29:01.518+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-35/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:01.518+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-35", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-35-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a65600df4929841901fd') } ntoreturn:1 keyUpdates:0 reslen:37 757ms 2014-12-23T17:29:04.039+0000 I INDEX [conn2] build index on: benchdb1.COL-36 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-36" } 2014-12-23T17:29:04.040+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:04.040+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:04.348+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-36 with requested shard version 1|2||5499a65c00df492984190202, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:04.349+0000 I SHARDING [conn2] collection benchdb1.COL-36 was previously unsharded, new metadata loaded with shard version 1|2||5499a65c00df492984190202 2014-12-23T17:29:04.349+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a65c00df492984190202, took 0ms 2014-12-23T17:29:04.388+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:04.389+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65c00df492984190202') } 2014-12-23T17:29:04.998+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6609a8f659f34bfb6f6 2014-12-23T17:29:04.998+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:04-5499a6609a8f659f34bfb6f7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355744998), what: "moveChunk.start", ns: "benchdb1.COL-36", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:05.064+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-36 based on current shard version 1|2||5499a65c00df492984190202, current metadata version is 1|2||5499a65c00df492984190202 2014-12-23T17:29:05.065+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-36 already up to date (shard version : 1|2||5499a65c00df492984190202, took 0ms) 2014-12-23T17:29:05.065+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a65c00df492984190202 2014-12-23T17:29:05.065+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:05.069+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:05.071+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:05.076+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:05.076+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:05.076+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:05.076+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a65c00df492984190202 2014-12-23T17:29:05.133+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:05.133+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a65c00df492984190202 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-36' 2014-12-23T17:29:05.438+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:05-5499a6619a8f659f34bfb6f8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355745438), what: "moveChunk.commit", ns: "benchdb1.COL-36", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:05.504+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:05.504+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:05.505+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-36 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3637 2014-12-23T17:29:05.505+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-36 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:05.505+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:05.808+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:05.808+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:05-5499a6619a8f659f34bfb6f9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355745808), what: "moveChunk.from", ns: "benchdb1.COL-36", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 676, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 428, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:05.864+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65c00df492984190202') } ntoreturn:1 keyUpdates:0 reslen:37 1476ms 2014-12-23T17:29:05.866+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:05.866+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65c00df492984190202') } 2014-12-23T17:29:06.473+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6619a8f659f34bfb6fa 2014-12-23T17:29:06.473+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:06-5499a6629a8f659f34bfb6fb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355746473), what: "moveChunk.start", ns: "benchdb1.COL-36", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:06.540+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-36 based on current shard version 2|0||5499a65c00df492984190202, current metadata version is 2|0||5499a65c00df492984190202 2014-12-23T17:29:06.541+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-36 from shard version 2|0||5499a65c00df492984190202 to shard version 2|1||5499a65c00df492984190202 2014-12-23T17:29:06.541+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a65c00df492984190202, took 0ms 2014-12-23T17:29:06.541+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a65c00df492984190202 2014-12-23T17:29:06.541+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:06.543+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:06.546+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:06.550+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:06.559+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:06.575+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:06.575+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:06.575+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:06.576+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a65c00df492984190202 2014-12-23T17:29:06.632+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:06.632+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a65c00df492984190202 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-36' 2014-12-23T17:29:06.937+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:06-5499a6629a8f659f34bfb6fc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355746937), what: "moveChunk.commit", ns: "benchdb1.COL-36", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:06.991+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:06.991+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:06.991+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-36 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3645 2014-12-23T17:29:06.991+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-36 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:06.991+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:07.294+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:07.294+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:07-5499a6639a8f659f34bfb6fd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355747294), what: "moveChunk.from", ns: "benchdb1.COL-36", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 674, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 415, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:07.353+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-36", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-36-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a65c00df492984190202') } ntoreturn:1 keyUpdates:0 reslen:37 1486ms 2014-12-23T17:29:07.355+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-36", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-36-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a65c00df492984190202') } 2014-12-23T17:29:07.961+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6639a8f659f34bfb6fe 2014-12-23T17:29:07.961+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-36 based on current shard version 3|0||5499a65c00df492984190202, current metadata version is 3|0||5499a65c00df492984190202 2014-12-23T17:29:07.962+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-36 from shard version 3|0||5499a65c00df492984190202 to shard version 3|1||5499a65c00df492984190202 2014-12-23T17:29:07.962+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a65c00df492984190202, took 0ms 2014-12-23T17:29:07.962+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a65c00df492984190202 2014-12-23T17:29:08.267+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:08-5499a6649a8f659f34bfb6ff", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355748267), what: "split", ns: "benchdb1.COL-36", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a65c00df492984190202') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a65c00df492984190202') } } } 2014-12-23T17:29:08.579+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-36/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:08.579+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-36", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-36-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a65c00df492984190202') } ntoreturn:1 keyUpdates:0 reslen:37 1224ms 2014-12-23T17:29:08.882+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:29:08 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:29:11.086+0000 I INDEX [conn2] build index on: benchdb1.COL-37 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-37" } 2014-12-23T17:29:11.086+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:11.087+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:11.430+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-37 with requested shard version 1|2||5499a66400df492984190205, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:11.431+0000 I SHARDING [conn2] collection benchdb1.COL-37 was previously unsharded, new metadata loaded with shard version 1|2||5499a66400df492984190205 2014-12-23T17:29:11.431+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a66400df492984190205, took 0ms 2014-12-23T17:29:11.473+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:11.473+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66400df492984190205') } 2014-12-23T17:29:12.001+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6679a8f659f34bfb701 2014-12-23T17:29:12.001+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:11-5499a6679a8f659f34bfb702", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355752001), what: "moveChunk.start", ns: "benchdb1.COL-37", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:12.071+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-37 based on current shard version 1|2||5499a66400df492984190205, current metadata version is 1|2||5499a66400df492984190205 2014-12-23T17:29:12.072+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-37 already up to date (shard version : 1|2||5499a66400df492984190205, took 0ms) 2014-12-23T17:29:12.072+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a66400df492984190205 2014-12-23T17:29:12.072+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:12.075+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:12.077+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:12.082+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:12.082+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:12.082+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:12.082+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a66400df492984190205 2014-12-23T17:29:12.141+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:12.141+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a66400df492984190205 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-37' 2014-12-23T17:29:12.370+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:12-5499a6689a8f659f34bfb703", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355752370), what: "moveChunk.commit", ns: "benchdb1.COL-37", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:12.430+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:12.430+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:12.430+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-37 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3664 2014-12-23T17:29:12.430+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-37 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:12.430+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:12.628+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:12.628+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:12-5499a6689a8f659f34bfb704", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355752628), what: "moveChunk.from", ns: "benchdb1.COL-37", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 598, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 348, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:12.689+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66400df492984190205') } ntoreturn:1 keyUpdates:0 reslen:37 1215ms 2014-12-23T17:29:12.690+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:12.691+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66400df492984190205') } 2014-12-23T17:29:12.983+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6689a8f659f34bfb705 2014-12-23T17:29:12.983+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:12-5499a6689a8f659f34bfb706", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355752983), what: "moveChunk.start", ns: "benchdb1.COL-37", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:13.035+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-37 based on current shard version 2|0||5499a66400df492984190205, current metadata version is 2|0||5499a66400df492984190205 2014-12-23T17:29:13.036+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-37 from shard version 2|0||5499a66400df492984190205 to shard version 2|1||5499a66400df492984190205 2014-12-23T17:29:13.036+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a66400df492984190205, took 0ms 2014-12-23T17:29:13.036+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a66400df492984190205 2014-12-23T17:29:13.036+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:13.039+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:13.041+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:13.045+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:13.054+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:13.070+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:13.070+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:13.071+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:13.071+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a66400df492984190205 2014-12-23T17:29:13.122+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:13.122+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a66400df492984190205 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-37' 2014-12-23T17:29:13.358+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:13-5499a6699a8f659f34bfb707", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355753358), what: "moveChunk.commit", ns: "benchdb1.COL-37", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:13.424+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:13.424+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:13.424+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-37 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3674 2014-12-23T17:29:13.424+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-37 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:13.424+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:13.599+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:13.599+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:13-5499a6699a8f659f34bfb708", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355753599), what: "moveChunk.from", ns: "benchdb1.COL-37", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 345, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 353, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:13.657+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-37", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-37-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66400df492984190205') } ntoreturn:1 keyUpdates:0 reslen:37 966ms 2014-12-23T17:29:13.659+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-37", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-37-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a66400df492984190205') } 2014-12-23T17:29:13.944+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6699a8f659f34bfb709 2014-12-23T17:29:13.944+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-37 based on current shard version 3|0||5499a66400df492984190205, current metadata version is 3|0||5499a66400df492984190205 2014-12-23T17:29:13.945+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-37 from shard version 3|0||5499a66400df492984190205 to shard version 3|1||5499a66400df492984190205 2014-12-23T17:29:13.945+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a66400df492984190205, took 0ms 2014-12-23T17:29:13.945+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a66400df492984190205 2014-12-23T17:29:14.217+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:14-5499a66a9a8f659f34bfb70a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355754217), what: "split", ns: "benchdb1.COL-37", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a66400df492984190205') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a66400df492984190205') } } } 2014-12-23T17:29:14.553+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-37/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:14.553+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-37", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-37-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a66400df492984190205') } ntoreturn:1 keyUpdates:0 reslen:37 893ms 2014-12-23T17:29:17.077+0000 I INDEX [conn2] build index on: benchdb1.COL-38 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-38" } 2014-12-23T17:29:17.077+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:17.078+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:17.423+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-38 with requested shard version 1|2||5499a66a00df49298419020a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:17.424+0000 I SHARDING [conn2] collection benchdb1.COL-38 was previously unsharded, new metadata loaded with shard version 1|2||5499a66a00df49298419020a 2014-12-23T17:29:17.424+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a66a00df49298419020a, took 0ms 2014-12-23T17:29:17.494+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:17.495+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66a00df49298419020a') } 2014-12-23T17:29:18.102+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a66d9a8f659f34bfb70c 2014-12-23T17:29:18.102+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:18-5499a66e9a8f659f34bfb70d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355758102), what: "moveChunk.start", ns: "benchdb1.COL-38", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:18.170+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-38 based on current shard version 1|2||5499a66a00df49298419020a, current metadata version is 1|2||5499a66a00df49298419020a 2014-12-23T17:29:18.171+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-38 already up to date (shard version : 1|2||5499a66a00df49298419020a, took 0ms) 2014-12-23T17:29:18.171+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a66a00df49298419020a 2014-12-23T17:29:18.172+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:18.174+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:18.177+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:18.181+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:18.181+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:18.182+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:18.182+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a66a00df49298419020a 2014-12-23T17:29:18.238+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:18.238+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a66a00df49298419020a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-38' 2014-12-23T17:29:18.543+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:18-5499a66e9a8f659f34bfb70e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355758543), what: "moveChunk.commit", ns: "benchdb1.COL-38", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:18.612+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:18.613+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:18.613+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-38 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3693 2014-12-23T17:29:18.613+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-38 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:18.613+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:18.916+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:18.917+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:18-5499a66e9a8f659f34bfb70f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355758917), what: "moveChunk.from", ns: "benchdb1.COL-38", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 676, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 431, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:18.971+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66a00df49298419020a') } ntoreturn:1 keyUpdates:0 reslen:37 1476ms 2014-12-23T17:29:18.972+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:18.973+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66a00df49298419020a') } 2014-12-23T17:29:19.580+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a66e9a8f659f34bfb710 2014-12-23T17:29:19.580+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:19-5499a66f9a8f659f34bfb711", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355759580), what: "moveChunk.start", ns: "benchdb1.COL-38", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:19.645+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-38 based on current shard version 2|0||5499a66a00df49298419020a, current metadata version is 2|0||5499a66a00df49298419020a 2014-12-23T17:29:19.646+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-38 from shard version 2|0||5499a66a00df49298419020a to shard version 2|1||5499a66a00df49298419020a 2014-12-23T17:29:19.646+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a66a00df49298419020a, took 0ms 2014-12-23T17:29:19.646+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a66a00df49298419020a 2014-12-23T17:29:19.646+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:19.649+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:19.651+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:19.655+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:19.664+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:19.680+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:19.680+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:19.681+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:19.681+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a66a00df49298419020a 2014-12-23T17:29:19.737+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:19.737+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a66a00df49298419020a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-38' 2014-12-23T17:29:20.019+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:20-5499a6709a8f659f34bfb712", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355760019), what: "moveChunk.commit", ns: "benchdb1.COL-38", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:20.074+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:20.074+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:20.075+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-38 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3701 2014-12-23T17:29:20.075+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-38 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:20.075+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:20.378+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:20.378+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:20-5499a6709a8f659f34bfb713", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355760378), what: "moveChunk.from", ns: "benchdb1.COL-38", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 394, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:20.437+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-38", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-38-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a66a00df49298419020a') } ntoreturn:1 keyUpdates:0 reslen:37 1464ms 2014-12-23T17:29:20.438+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-38", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-38-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a66a00df49298419020a') } 2014-12-23T17:29:21.032+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6709a8f659f34bfb714 2014-12-23T17:29:21.032+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-38 based on current shard version 3|0||5499a66a00df49298419020a, current metadata version is 3|0||5499a66a00df49298419020a 2014-12-23T17:29:21.033+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-38 from shard version 3|0||5499a66a00df49298419020a to shard version 3|1||5499a66a00df49298419020a 2014-12-23T17:29:21.033+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a66a00df49298419020a, took 0ms 2014-12-23T17:29:21.033+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a66a00df49298419020a 2014-12-23T17:29:21.337+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:21-5499a6719a8f659f34bfb715", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355761337), what: "split", ns: "benchdb1.COL-38", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a66a00df49298419020a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a66a00df49298419020a') } } } 2014-12-23T17:29:21.720+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-38/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:21.720+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-38", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-38-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a66a00df49298419020a') } ntoreturn:1 keyUpdates:0 reslen:37 1281ms 2014-12-23T17:29:24.226+0000 I INDEX [conn2] build index on: benchdb1.COL-39 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-39" } 2014-12-23T17:29:24.226+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:24.227+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:24.570+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-39 with requested shard version 1|2||5499a67100df49298419020e, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:24.570+0000 I SHARDING [conn2] collection benchdb1.COL-39 was previously unsharded, new metadata loaded with shard version 1|2||5499a67100df49298419020e 2014-12-23T17:29:24.571+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a67100df49298419020e, took 0ms 2014-12-23T17:29:24.610+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:24.611+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67100df49298419020e') } 2014-12-23T17:29:24.930+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6749a8f659f34bfb717 2014-12-23T17:29:24.930+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:24-5499a6749a8f659f34bfb718", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355764930), what: "moveChunk.start", ns: "benchdb1.COL-39", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:24.981+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-39 based on current shard version 1|2||5499a67100df49298419020e, current metadata version is 1|2||5499a67100df49298419020e 2014-12-23T17:29:24.982+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-39 already up to date (shard version : 1|2||5499a67100df49298419020e, took 0ms) 2014-12-23T17:29:24.982+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a67100df49298419020e 2014-12-23T17:29:24.982+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:24.985+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:24.987+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:24.992+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:24.992+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:24.992+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:24.992+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a67100df49298419020e 2014-12-23T17:29:25.042+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:25.042+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a67100df49298419020e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-39' 2014-12-23T17:29:25.242+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:25-5499a6759a8f659f34bfb719", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355765242), what: "moveChunk.commit", ns: "benchdb1.COL-39", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:25.291+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:25.292+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:25.292+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-39 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3721 2014-12-23T17:29:25.292+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-39 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:25.292+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:25.461+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:25.461+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:25-5499a6759a8f659f34bfb71a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355765461), what: "moveChunk.from", ns: "benchdb1.COL-39", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 371, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 299, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:25.517+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67100df49298419020e') } ntoreturn:1 keyUpdates:0 reslen:37 907ms 2014-12-23T17:29:25.519+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:25.519+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67100df49298419020e') } 2014-12-23T17:29:25.819+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6759a8f659f34bfb71b 2014-12-23T17:29:25.819+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:25-5499a6759a8f659f34bfb71c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355765819), what: "moveChunk.start", ns: "benchdb1.COL-39", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:25.918+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-39 based on current shard version 2|0||5499a67100df49298419020e, current metadata version is 2|0||5499a67100df49298419020e 2014-12-23T17:29:25.919+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-39 from shard version 2|0||5499a67100df49298419020e to shard version 2|1||5499a67100df49298419020e 2014-12-23T17:29:25.919+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a67100df49298419020e, took 0ms 2014-12-23T17:29:25.919+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a67100df49298419020e 2014-12-23T17:29:25.919+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:25.921+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:25.924+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:25.928+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:25.936+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:25.953+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:25.953+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:25.953+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:25.953+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a67100df49298419020e 2014-12-23T17:29:25.993+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:25.994+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a67100df49298419020e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-39' 2014-12-23T17:29:26.192+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:26-5499a6769a8f659f34bfb71d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355766191), what: "moveChunk.commit", ns: "benchdb1.COL-39", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:26.244+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:26.244+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:26.244+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-39 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3730 2014-12-23T17:29:26.244+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-39 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:26.244+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:26.479+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:26.479+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:26-5499a6769a8f659f34bfb71e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355766479), what: "moveChunk.from", ns: "benchdb1.COL-39", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 399, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 291, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:26.536+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-39", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-39-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67100df49298419020e') } ntoreturn:1 keyUpdates:0 reslen:37 1017ms 2014-12-23T17:29:26.537+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-39", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-39-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a67100df49298419020e') } 2014-12-23T17:29:27.144+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6769a8f659f34bfb71f 2014-12-23T17:29:27.144+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-39 based on current shard version 3|0||5499a67100df49298419020e, current metadata version is 3|0||5499a67100df49298419020e 2014-12-23T17:29:27.144+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-39 from shard version 3|0||5499a67100df49298419020e to shard version 3|1||5499a67100df49298419020e 2014-12-23T17:29:27.144+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a67100df49298419020e, took 0ms 2014-12-23T17:29:27.144+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a67100df49298419020e 2014-12-23T17:29:27.449+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:27-5499a6779a8f659f34bfb720", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355767449), what: "split", ns: "benchdb1.COL-39", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a67100df49298419020e') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a67100df49298419020e') } } } 2014-12-23T17:29:27.865+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-39/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:27.865+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-39", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-39-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a67100df49298419020e') } ntoreturn:1 keyUpdates:0 reslen:37 1327ms 2014-12-23T17:29:30.409+0000 I INDEX [conn2] build index on: benchdb1.COL-40 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-40" } 2014-12-23T17:29:30.409+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:30.410+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:30.702+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-40 with requested shard version 1|2||5499a67700df492984190212, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:30.702+0000 I SHARDING [conn2] collection benchdb1.COL-40 was previously unsharded, new metadata loaded with shard version 1|2||5499a67700df492984190212 2014-12-23T17:29:30.702+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a67700df492984190212, took 0ms 2014-12-23T17:29:30.746+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:30.747+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67700df492984190212') } 2014-12-23T17:29:31.353+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a67a9a8f659f34bfb722 2014-12-23T17:29:31.353+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:31-5499a67b9a8f659f34bfb723", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355771353), what: "moveChunk.start", ns: "benchdb1.COL-40", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:31.414+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-40 based on current shard version 1|2||5499a67700df492984190212, current metadata version is 1|2||5499a67700df492984190212 2014-12-23T17:29:31.415+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-40 already up to date (shard version : 1|2||5499a67700df492984190212, took 0ms) 2014-12-23T17:29:31.415+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a67700df492984190212 2014-12-23T17:29:31.415+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:31.418+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:31.420+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:31.424+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:31.424+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:31.425+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:31.425+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a67700df492984190212 2014-12-23T17:29:31.481+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:31.481+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a67700df492984190212 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-40' 2014-12-23T17:29:31.786+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:31-5499a67b9a8f659f34bfb724", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355771786), what: "moveChunk.commit", ns: "benchdb1.COL-40", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:31.850+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:31.850+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:31.850+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-40 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3748 2014-12-23T17:29:31.850+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-40 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:31.850+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:32.153+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:32.154+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:32-5499a67c9a8f659f34bfb725", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355772154), what: "moveChunk.from", ns: "benchdb1.COL-40", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 425, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:32.212+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67700df492984190212') } ntoreturn:1 keyUpdates:0 reslen:37 1465ms 2014-12-23T17:29:32.213+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:32.214+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67700df492984190212') } 2014-12-23T17:29:32.821+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a67c9a8f659f34bfb726 2014-12-23T17:29:32.821+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:32-5499a67c9a8f659f34bfb727", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355772821), what: "moveChunk.start", ns: "benchdb1.COL-40", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:32.882+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-40 based on current shard version 2|0||5499a67700df492984190212, current metadata version is 2|0||5499a67700df492984190212 2014-12-23T17:29:32.883+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-40 from shard version 2|0||5499a67700df492984190212 to shard version 2|1||5499a67700df492984190212 2014-12-23T17:29:32.883+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a67700df492984190212, took 0ms 2014-12-23T17:29:32.883+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a67700df492984190212 2014-12-23T17:29:32.883+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:32.885+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:32.888+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:32.892+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:32.900+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:32.917+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:32.917+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:32.917+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:32.917+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a67700df492984190212 2014-12-23T17:29:32.973+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:32.973+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a67700df492984190212 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-40' 2014-12-23T17:29:33.277+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:33-5499a67d9a8f659f34bfb728", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355773277), what: "moveChunk.commit", ns: "benchdb1.COL-40", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:33.330+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:33.330+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:33.330+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-40 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3756 2014-12-23T17:29:33.330+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-40 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:33.330+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:33.633+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:33.633+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:33-5499a67d9a8f659f34bfb729", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355773633), what: "moveChunk.from", ns: "benchdb1.COL-40", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 413, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:33.687+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-40", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-40-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67700df492984190212') } ntoreturn:1 keyUpdates:0 reslen:37 1473ms 2014-12-23T17:29:33.689+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-40", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-40-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a67700df492984190212') } 2014-12-23T17:29:34.295+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a67d9a8f659f34bfb72a 2014-12-23T17:29:34.295+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-40 based on current shard version 3|0||5499a67700df492984190212, current metadata version is 3|0||5499a67700df492984190212 2014-12-23T17:29:34.296+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-40 from shard version 3|0||5499a67700df492984190212 to shard version 3|1||5499a67700df492984190212 2014-12-23T17:29:34.296+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a67700df492984190212, took 0ms 2014-12-23T17:29:34.296+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a67700df492984190212 2014-12-23T17:29:34.600+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:34-5499a67e9a8f659f34bfb72b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355774600), what: "split", ns: "benchdb1.COL-40", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a67700df492984190212') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a67700df492984190212') } } } 2014-12-23T17:29:34.980+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-40/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:34.980+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-40", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-40-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a67700df492984190212') } ntoreturn:1 keyUpdates:0 reslen:37 1291ms 2014-12-23T17:29:37.306+0000 I INDEX [conn2] build index on: benchdb1.COL-41 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-41" } 2014-12-23T17:29:37.306+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:37.307+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:37.604+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-41 with requested shard version 1|2||5499a67e00df492984190216, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:37.605+0000 I SHARDING [conn2] collection benchdb1.COL-41 was previously unsharded, new metadata loaded with shard version 1|2||5499a67e00df492984190216 2014-12-23T17:29:37.605+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a67e00df492984190216, took 0ms 2014-12-23T17:29:37.652+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:37.653+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67e00df492984190216') } 2014-12-23T17:29:37.930+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6819a8f659f34bfb72d 2014-12-23T17:29:37.930+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:37-5499a6819a8f659f34bfb72e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355777930), what: "moveChunk.start", ns: "benchdb1.COL-41", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:37.984+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-41 based on current shard version 1|2||5499a67e00df492984190216, current metadata version is 1|2||5499a67e00df492984190216 2014-12-23T17:29:37.985+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-41 already up to date (shard version : 1|2||5499a67e00df492984190216, took 0ms) 2014-12-23T17:29:37.985+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a67e00df492984190216 2014-12-23T17:29:37.985+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:37.988+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:37.991+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:37.995+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:37.995+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:37.995+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:37.995+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a67e00df492984190216 2014-12-23T17:29:38.098+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:38.098+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a67e00df492984190216 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-41' 2014-12-23T17:29:38.268+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:38-5499a6829a8f659f34bfb72f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355778268), what: "moveChunk.commit", ns: "benchdb1.COL-41", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:38.323+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:38.323+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:38.323+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-41 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3777 2014-12-23T17:29:38.323+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-41 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:38.323+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:38.489+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:38.489+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:38-5499a6829a8f659f34bfb730", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355778489), what: "moveChunk.from", ns: "benchdb1.COL-41", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 332, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 327, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:38.544+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67e00df492984190216') } ntoreturn:1 keyUpdates:0 reslen:37 891ms 2014-12-23T17:29:38.545+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:38.545+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67e00df492984190216') } 2014-12-23T17:29:38.848+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6829a8f659f34bfb731 2014-12-23T17:29:38.848+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:38-5499a6829a8f659f34bfb732", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355778848), what: "moveChunk.start", ns: "benchdb1.COL-41", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:38.903+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-41 based on current shard version 2|0||5499a67e00df492984190216, current metadata version is 2|0||5499a67e00df492984190216 2014-12-23T17:29:38.903+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-41 from shard version 2|0||5499a67e00df492984190216 to shard version 2|1||5499a67e00df492984190216 2014-12-23T17:29:38.903+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a67e00df492984190216, took 0ms 2014-12-23T17:29:38.903+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a67e00df492984190216 2014-12-23T17:29:38.904+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:38.906+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:38.908+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:38.913+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:38.921+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:38.938+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:38.938+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:38.938+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:38.938+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a67e00df492984190216 2014-12-23T17:29:39.002+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:39.002+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a67e00df492984190216 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-41' 2014-12-23T17:29:39.241+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:29:38 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:29:39.242+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:39-5499a6839a8f659f34bfb733", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355779242), what: "moveChunk.commit", ns: "benchdb1.COL-41", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:39.310+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:39.310+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:39.310+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-41 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3785 2014-12-23T17:29:39.310+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-41 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:39.310+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:39.613+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:39.613+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:39-5499a6839a8f659f34bfb734", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355779613), what: "moveChunk.from", ns: "benchdb1.COL-41", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 357, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 372, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:39.683+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-41", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-41-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a67e00df492984190216') } ntoreturn:1 keyUpdates:0 reslen:37 1137ms 2014-12-23T17:29:39.684+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-41", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-41-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a67e00df492984190216') } 2014-12-23T17:29:40.291+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6839a8f659f34bfb735 2014-12-23T17:29:40.291+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-41 based on current shard version 3|0||5499a67e00df492984190216, current metadata version is 3|0||5499a67e00df492984190216 2014-12-23T17:29:40.292+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-41 from shard version 3|0||5499a67e00df492984190216 to shard version 3|1||5499a67e00df492984190216 2014-12-23T17:29:40.292+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a67e00df492984190216, took 0ms 2014-12-23T17:29:40.292+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a67e00df492984190216 2014-12-23T17:29:40.596+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:40-5499a6849a8f659f34bfb736", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355780596), what: "split", ns: "benchdb1.COL-41", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a67e00df492984190216') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a67e00df492984190216') } } } 2014-12-23T17:29:40.928+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-41/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:40.928+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-41", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-41-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a67e00df492984190216') } ntoreturn:1 keyUpdates:0 reslen:37 1243ms 2014-12-23T17:29:43.435+0000 I INDEX [conn2] build index on: benchdb1.COL-42 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-42" } 2014-12-23T17:29:43.435+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:43.435+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:43.725+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-42 with requested shard version 1|2||5499a68400df49298419021a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:43.726+0000 I SHARDING [conn2] collection benchdb1.COL-42 was previously unsharded, new metadata loaded with shard version 1|2||5499a68400df49298419021a 2014-12-23T17:29:43.726+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a68400df49298419021a, took 0ms 2014-12-23T17:29:43.768+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:43.768+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68400df49298419021a') } 2014-12-23T17:29:44.374+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6879a8f659f34bfb738 2014-12-23T17:29:44.374+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:44-5499a6889a8f659f34bfb739", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355784374), what: "moveChunk.start", ns: "benchdb1.COL-42", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:44.441+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-42 based on current shard version 1|2||5499a68400df49298419021a, current metadata version is 1|2||5499a68400df49298419021a 2014-12-23T17:29:44.442+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-42 already up to date (shard version : 1|2||5499a68400df49298419021a, took 0ms) 2014-12-23T17:29:44.442+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a68400df49298419021a 2014-12-23T17:29:44.442+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:44.445+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:44.447+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:44.452+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:44.452+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:44.452+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:44.452+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a68400df49298419021a 2014-12-23T17:29:44.511+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:44.511+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a68400df49298419021a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-42' 2014-12-23T17:29:44.815+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:44-5499a6889a8f659f34bfb73a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355784815), what: "moveChunk.commit", ns: "benchdb1.COL-42", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:44.873+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:44.873+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:44.874+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-42 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3804 2014-12-23T17:29:44.874+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-42 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:44.874+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:45.127+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:45.127+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:45-5499a6899a8f659f34bfb73b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355785127), what: "moveChunk.from", ns: "benchdb1.COL-42", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 421, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:45.185+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68400df49298419021a') } ntoreturn:1 keyUpdates:0 reslen:37 1417ms 2014-12-23T17:29:45.186+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:45.187+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68400df49298419021a') } 2014-12-23T17:29:45.793+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6899a8f659f34bfb73c 2014-12-23T17:29:45.793+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:45-5499a6899a8f659f34bfb73d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355785793), what: "moveChunk.start", ns: "benchdb1.COL-42", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:45.865+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-42 based on current shard version 2|0||5499a68400df49298419021a, current metadata version is 2|0||5499a68400df49298419021a 2014-12-23T17:29:45.866+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-42 from shard version 2|0||5499a68400df49298419021a to shard version 2|1||5499a68400df49298419021a 2014-12-23T17:29:45.866+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a68400df49298419021a, took 0ms 2014-12-23T17:29:45.866+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a68400df49298419021a 2014-12-23T17:29:45.866+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:45.869+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:45.871+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:45.876+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:45.884+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:45.900+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:45.900+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:45.901+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:45.901+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a68400df49298419021a 2014-12-23T17:29:45.951+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:45.951+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a68400df49298419021a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-42' 2014-12-23T17:29:46.256+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:46-5499a68a9a8f659f34bfb73e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355786256), what: "moveChunk.commit", ns: "benchdb1.COL-42", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:46.324+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:46.324+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:46.325+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-42 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3812 2014-12-23T17:29:46.325+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-42 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:46.325+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:46.580+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:46.580+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:46-5499a68a9a8f659f34bfb73f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355786580), what: "moveChunk.from", ns: "benchdb1.COL-42", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 679, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:46.644+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-42", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-42-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68400df49298419021a') } ntoreturn:1 keyUpdates:0 reslen:37 1458ms 2014-12-23T17:29:46.646+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-42", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-42-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a68400df49298419021a') } 2014-12-23T17:29:47.252+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a68a9a8f659f34bfb740 2014-12-23T17:29:47.252+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-42 based on current shard version 3|0||5499a68400df49298419021a, current metadata version is 3|0||5499a68400df49298419021a 2014-12-23T17:29:47.253+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-42 from shard version 3|0||5499a68400df49298419021a to shard version 3|1||5499a68400df49298419021a 2014-12-23T17:29:47.253+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a68400df49298419021a, took 0ms 2014-12-23T17:29:47.253+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a68400df49298419021a 2014-12-23T17:29:47.558+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:47-5499a68b9a8f659f34bfb741", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355787558), what: "split", ns: "benchdb1.COL-42", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a68400df49298419021a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a68400df49298419021a') } } } 2014-12-23T17:29:47.936+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-42/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:47.936+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-42", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-42-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a68400df49298419021a') } ntoreturn:1 keyUpdates:0 reslen:37 1290ms 2014-12-23T17:29:50.151+0000 I INDEX [conn2] build index on: benchdb1.COL-43 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-43" } 2014-12-23T17:29:50.151+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:50.152+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:50.439+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-43 with requested shard version 1|2||5499a68b00df49298419021e, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:50.440+0000 I SHARDING [conn2] collection benchdb1.COL-43 was previously unsharded, new metadata loaded with shard version 1|2||5499a68b00df49298419021e 2014-12-23T17:29:50.440+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a68b00df49298419021e, took 0ms 2014-12-23T17:29:50.484+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:50.484+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68b00df49298419021e') } 2014-12-23T17:29:50.779+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a68e9a8f659f34bfb743 2014-12-23T17:29:50.779+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:50-5499a68e9a8f659f34bfb744", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355790779), what: "moveChunk.start", ns: "benchdb1.COL-43", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:50.832+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-43 based on current shard version 1|2||5499a68b00df49298419021e, current metadata version is 1|2||5499a68b00df49298419021e 2014-12-23T17:29:50.833+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-43 already up to date (shard version : 1|2||5499a68b00df49298419021e, took 0ms) 2014-12-23T17:29:50.833+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a68b00df49298419021e 2014-12-23T17:29:50.833+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:50.836+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:50.838+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:50.843+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:50.843+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:50.843+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:50.843+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a68b00df49298419021e 2014-12-23T17:29:50.889+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:50.889+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a68b00df49298419021e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-43' 2014-12-23T17:29:51.060+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:51-5499a68f9a8f659f34bfb745", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355791060), what: "moveChunk.commit", ns: "benchdb1.COL-43", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:51.118+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:51.118+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:51.118+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-43 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3832 2014-12-23T17:29:51.118+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-43 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:51.119+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:51.289+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:51.289+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:51-5499a68f9a8f659f34bfb746", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355791289), what: "moveChunk.from", ns: "benchdb1.COL-43", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 348, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 275, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:51.336+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68b00df49298419021e') } ntoreturn:1 keyUpdates:0 reslen:37 852ms 2014-12-23T17:29:51.338+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:51.338+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68b00df49298419021e') } 2014-12-23T17:29:51.639+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a68f9a8f659f34bfb747 2014-12-23T17:29:51.639+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:51-5499a68f9a8f659f34bfb748", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355791639), what: "moveChunk.start", ns: "benchdb1.COL-43", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:51.692+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-43 based on current shard version 2|0||5499a68b00df49298419021e, current metadata version is 2|0||5499a68b00df49298419021e 2014-12-23T17:29:51.693+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-43 from shard version 2|0||5499a68b00df49298419021e to shard version 2|1||5499a68b00df49298419021e 2014-12-23T17:29:51.693+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a68b00df49298419021e, took 0ms 2014-12-23T17:29:51.693+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a68b00df49298419021e 2014-12-23T17:29:51.693+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:51.695+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:51.698+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:51.702+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:51.711+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:51.727+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:51.727+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:51.727+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:51.727+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a68b00df49298419021e 2014-12-23T17:29:51.774+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:51.774+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a68b00df49298419021e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-43' 2014-12-23T17:29:52.004+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:52-5499a6909a8f659f34bfb749", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355792004), what: "moveChunk.commit", ns: "benchdb1.COL-43", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:52.049+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:52.049+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:52.049+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-43 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3840 2014-12-23T17:29:52.050+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-43 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:52.050+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:52.306+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:52.306+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:52-5499a6909a8f659f34bfb74a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355792306), what: "moveChunk.from", ns: "benchdb1.COL-43", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 354, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 322, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:52.363+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-43", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-43-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a68b00df49298419021e') } ntoreturn:1 keyUpdates:0 reslen:37 1025ms 2014-12-23T17:29:52.366+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-43", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-43-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a68b00df49298419021e') } 2014-12-23T17:29:52.973+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6909a8f659f34bfb74b 2014-12-23T17:29:52.973+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-43 based on current shard version 3|0||5499a68b00df49298419021e, current metadata version is 3|0||5499a68b00df49298419021e 2014-12-23T17:29:52.973+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-43 from shard version 3|0||5499a68b00df49298419021e to shard version 3|1||5499a68b00df49298419021e 2014-12-23T17:29:52.974+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a68b00df49298419021e, took 0ms 2014-12-23T17:29:52.974+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a68b00df49298419021e 2014-12-23T17:29:53.278+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:53-5499a6919a8f659f34bfb74c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355793278), what: "split", ns: "benchdb1.COL-43", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a68b00df49298419021e') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a68b00df49298419021e') } } } 2014-12-23T17:29:53.651+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-43/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:53.651+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-43", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-43-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a68b00df49298419021e') } ntoreturn:1 keyUpdates:0 reslen:37 1285ms 2014-12-23T17:29:56.241+0000 I INDEX [conn2] build index on: benchdb1.COL-44 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-44" } 2014-12-23T17:29:56.241+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:29:56.242+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:29:56.550+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-44 with requested shard version 1|2||5499a69100df492984190222, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:29:56.550+0000 I SHARDING [conn2] collection benchdb1.COL-44 was previously unsharded, new metadata loaded with shard version 1|2||5499a69100df492984190222 2014-12-23T17:29:56.551+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a69100df492984190222, took 0ms 2014-12-23T17:29:56.599+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:56.599+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69100df492984190222') } 2014-12-23T17:29:57.206+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6949a8f659f34bfb74e 2014-12-23T17:29:57.206+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:57-5499a6959a8f659f34bfb74f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355797206), what: "moveChunk.start", ns: "benchdb1.COL-44", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:29:57.265+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-44 based on current shard version 1|2||5499a69100df492984190222, current metadata version is 1|2||5499a69100df492984190222 2014-12-23T17:29:57.266+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-44 already up to date (shard version : 1|2||5499a69100df492984190222, took 0ms) 2014-12-23T17:29:57.266+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a69100df492984190222 2014-12-23T17:29:57.266+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:57.269+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:57.271+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:57.276+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:57.276+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:57.276+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:57.276+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a69100df492984190222 2014-12-23T17:29:57.335+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:57.335+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a69100df492984190222 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-44' 2014-12-23T17:29:57.639+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:57-5499a6959a8f659f34bfb750", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355797639), what: "moveChunk.commit", ns: "benchdb1.COL-44", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:57.703+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:57.703+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:57.703+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-44 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3859 2014-12-23T17:29:57.703+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-44 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:29:57.703+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:58.006+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:58.006+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:58-5499a6969a8f659f34bfb751", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355798006), what: "moveChunk.from", ns: "benchdb1.COL-44", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 427, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:29:58.071+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69100df492984190222') } ntoreturn:1 keyUpdates:0 reslen:37 1472ms 2014-12-23T17:29:58.074+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:29:58.074+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69100df492984190222') } 2014-12-23T17:29:58.681+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6969a8f659f34bfb752 2014-12-23T17:29:58.681+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:58-5499a6969a8f659f34bfb753", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355798681), what: "moveChunk.start", ns: "benchdb1.COL-44", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:29:58.744+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-44 based on current shard version 2|0||5499a69100df492984190222, current metadata version is 2|0||5499a69100df492984190222 2014-12-23T17:29:58.745+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-44 from shard version 2|0||5499a69100df492984190222 to shard version 2|1||5499a69100df492984190222 2014-12-23T17:29:58.745+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a69100df492984190222, took 0ms 2014-12-23T17:29:58.745+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a69100df492984190222 2014-12-23T17:29:58.745+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:29:58.747+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:58.750+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:58.754+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:58.762+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:58.779+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:29:58.779+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:29:58.779+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:29:58.779+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a69100df492984190222 2014-12-23T17:29:58.835+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:29:58.835+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a69100df492984190222 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-44' 2014-12-23T17:29:59.140+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:59-5499a6979a8f659f34bfb754", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355799140), what: "moveChunk.commit", ns: "benchdb1.COL-44", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:29:59.196+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:59.196+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:29:59.196+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-44 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3867 2014-12-23T17:29:59.196+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-44 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:29:59.196+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:29:59.499+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:29:59.499+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:29:59-5499a6979a8f659f34bfb755", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355799499), what: "moveChunk.from", ns: "benchdb1.COL-44", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 416, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:29:59.560+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-44", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-44-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69100df492984190222') } ntoreturn:1 keyUpdates:0 reslen:37 1486ms 2014-12-23T17:29:59.562+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-44", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-44-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a69100df492984190222') } 2014-12-23T17:30:00.123+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6979a8f659f34bfb756 2014-12-23T17:30:00.123+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-44 based on current shard version 3|0||5499a69100df492984190222, current metadata version is 3|0||5499a69100df492984190222 2014-12-23T17:30:00.124+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-44 from shard version 3|0||5499a69100df492984190222 to shard version 3|1||5499a69100df492984190222 2014-12-23T17:30:00.124+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a69100df492984190222, took 0ms 2014-12-23T17:30:00.124+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a69100df492984190222 2014-12-23T17:30:00.428+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:00-5499a6989a8f659f34bfb757", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419355800428), what: "split", ns: "benchdb1.COL-44", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a69100df492984190222') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a69100df492984190222') } } } 2014-12-23T17:30:00.809+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-44/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:00.809+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-44", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-44-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a69100df492984190222') } ntoreturn:1 keyUpdates:0 reslen:37 1247ms 2014-12-23T17:30:02.988+0000 I INDEX [conn2] build index on: benchdb1.COL-45 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-45" } 2014-12-23T17:30:02.988+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:02.989+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:03.321+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-45 with requested shard version 1|2||5499a69700df492984190226, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:03.322+0000 I SHARDING [conn2] collection benchdb1.COL-45 was previously unsharded, new metadata loaded with shard version 1|2||5499a69700df492984190226 2014-12-23T17:30:03.322+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a69700df492984190226, took 1ms 2014-12-23T17:30:03.430+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:03.430+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69700df492984190226') } 2014-12-23T17:30:03.738+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a69b9a8f659f34bfb759 2014-12-23T17:30:03.738+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:03-5499a69b9a8f659f34bfb75a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355803738), what: "moveChunk.start", ns: "benchdb1.COL-45", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:03.790+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-45 based on current shard version 1|2||5499a69700df492984190226, current metadata version is 1|2||5499a69700df492984190226 2014-12-23T17:30:03.791+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-45 already up to date (shard version : 1|2||5499a69700df492984190226, took 0ms) 2014-12-23T17:30:03.791+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a69700df492984190226 2014-12-23T17:30:03.792+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:03.795+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:03.797+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:03.801+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:03.802+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:03.802+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:03.802+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a69700df492984190226 2014-12-23T17:30:03.909+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:03.909+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a69700df492984190226 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-45' 2014-12-23T17:30:04.106+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:04-5499a69c9a8f659f34bfb75b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355804106), what: "moveChunk.commit", ns: "benchdb1.COL-45", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:04.169+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:04.169+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:04.169+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-45 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3888 2014-12-23T17:30:04.170+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-45 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:04.170+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:04.346+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:04.346+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:04-5499a69c9a8f659f34bfb75c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355804346), what: "moveChunk.from", ns: "benchdb1.COL-45", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 360, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 367, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:04.410+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69700df492984190226') } ntoreturn:1 keyUpdates:0 reslen:37 980ms 2014-12-23T17:30:04.411+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:04.412+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69700df492984190226') } 2014-12-23T17:30:04.689+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a69c9a8f659f34bfb75d 2014-12-23T17:30:04.689+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:04-5499a69c9a8f659f34bfb75e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355804689), what: "moveChunk.start", ns: "benchdb1.COL-45", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:04.750+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-45 based on current shard version 2|0||5499a69700df492984190226, current metadata version is 2|0||5499a69700df492984190226 2014-12-23T17:30:04.751+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-45 from shard version 2|0||5499a69700df492984190226 to shard version 2|1||5499a69700df492984190226 2014-12-23T17:30:04.751+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a69700df492984190226, took 0ms 2014-12-23T17:30:04.751+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a69700df492984190226 2014-12-23T17:30:04.751+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:04.754+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:04.756+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:04.761+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:04.769+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:04.786+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:04.786+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:04.786+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:04.786+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a69700df492984190226 2014-12-23T17:30:04.855+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:04.855+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a69700df492984190226 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-45' 2014-12-23T17:30:05.088+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:05-5499a69d9a8f659f34bfb75f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355805088), what: "moveChunk.commit", ns: "benchdb1.COL-45", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:05.146+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:05.146+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:05.146+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-45 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3896 2014-12-23T17:30:05.146+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-45 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:05.146+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:05.414+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:05.414+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:05-5499a69d9a8f659f34bfb760", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355805414), what: "moveChunk.from", ns: "benchdb1.COL-45", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 339, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 359, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:05.474+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-45", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-45-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69700df492984190226') } ntoreturn:1 keyUpdates:0 reslen:37 1062ms 2014-12-23T17:30:05.476+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-45", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-45-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a69700df492984190226') } 2014-12-23T17:30:06.083+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a69d9a8f659f34bfb761 2014-12-23T17:30:06.083+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-45 based on current shard version 3|0||5499a69700df492984190226, current metadata version is 3|0||5499a69700df492984190226 2014-12-23T17:30:06.084+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-45 from shard version 3|0||5499a69700df492984190226 to shard version 3|1||5499a69700df492984190226 2014-12-23T17:30:06.084+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a69700df492984190226, took 0ms 2014-12-23T17:30:06.084+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a69700df492984190226 2014-12-23T17:30:06.389+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:06-5499a69e9a8f659f34bfb762", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355806389), what: "split", ns: "benchdb1.COL-45", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a69700df492984190226') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a69700df492984190226') } } } 2014-12-23T17:30:06.776+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-45/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:06.776+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-45", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-45-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a69700df492984190226') } ntoreturn:1 keyUpdates:0 reslen:37 1300ms 2014-12-23T17:30:09.360+0000 I INDEX [conn2] build index on: benchdb1.COL-46 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-46" } 2014-12-23T17:30:09.360+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:09.361+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:09.647+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:30:09 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:30:09.690+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-46 with requested shard version 1|2||5499a69e00df49298419022a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:09.691+0000 I SHARDING [conn2] collection benchdb1.COL-46 was previously unsharded, new metadata loaded with shard version 1|2||5499a69e00df49298419022a 2014-12-23T17:30:09.691+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a69e00df49298419022a, took 0ms 2014-12-23T17:30:09.737+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:09.738+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69e00df49298419022a') } 2014-12-23T17:30:10.331+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6a19a8f659f34bfb764 2014-12-23T17:30:10.331+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:10-5499a6a29a8f659f34bfb765", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355810331), what: "moveChunk.start", ns: "benchdb1.COL-46", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:10.400+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-46 based on current shard version 1|2||5499a69e00df49298419022a, current metadata version is 1|2||5499a69e00df49298419022a 2014-12-23T17:30:10.401+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-46 already up to date (shard version : 1|2||5499a69e00df49298419022a, took 0ms) 2014-12-23T17:30:10.401+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a69e00df49298419022a 2014-12-23T17:30:10.401+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:10.404+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:10.406+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:10.411+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:10.411+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:10.411+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:10.411+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a69e00df49298419022a 2014-12-23T17:30:10.468+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:10.468+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a69e00df49298419022a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-46' 2014-12-23T17:30:10.729+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:10-5499a6a29a8f659f34bfb766", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355810729), what: "moveChunk.commit", ns: "benchdb1.COL-46", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:10.787+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:10.788+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:10.788+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-46 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3915 2014-12-23T17:30:10.788+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-46 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:10.788+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:11.091+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:11.091+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:11-5499a6a39a8f659f34bfb767", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355811091), what: "moveChunk.from", ns: "benchdb1.COL-46", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 376, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:11.165+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69e00df49298419022a') } ntoreturn:1 keyUpdates:0 reslen:37 1428ms 2014-12-23T17:30:11.167+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:11.167+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69e00df49298419022a') } 2014-12-23T17:30:11.728+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6a39a8f659f34bfb768 2014-12-23T17:30:11.728+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:11-5499a6a39a8f659f34bfb769", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355811728), what: "moveChunk.start", ns: "benchdb1.COL-46", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:11.800+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-46 based on current shard version 2|0||5499a69e00df49298419022a, current metadata version is 2|0||5499a69e00df49298419022a 2014-12-23T17:30:11.801+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-46 from shard version 2|0||5499a69e00df49298419022a to shard version 2|1||5499a69e00df49298419022a 2014-12-23T17:30:11.801+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a69e00df49298419022a, took 0ms 2014-12-23T17:30:11.801+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a69e00df49298419022a 2014-12-23T17:30:11.801+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:11.803+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:11.806+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:11.811+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:11.819+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:11.836+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:11.836+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:11.836+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:11.836+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a69e00df49298419022a 2014-12-23T17:30:11.900+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:11.900+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a69e00df49298419022a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-46' 2014-12-23T17:30:12.206+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:12-5499a6a49a8f659f34bfb76a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355812206), what: "moveChunk.commit", ns: "benchdb1.COL-46", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:12.262+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:12.262+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:12.262+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-46 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3923 2014-12-23T17:30:12.262+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-46 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:12.262+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:12.565+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:12.565+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:12-5499a6a49a8f659f34bfb76b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355812565), what: "moveChunk.from", ns: "benchdb1.COL-46", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 633, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 426, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:12.657+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-46", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-46-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a69e00df49298419022a') } ntoreturn:1 keyUpdates:0 reslen:37 1490ms 2014-12-23T17:30:12.659+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-46", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-46-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a69e00df49298419022a') } 2014-12-23T17:30:13.267+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6a49a8f659f34bfb76c 2014-12-23T17:30:13.267+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-46 based on current shard version 3|0||5499a69e00df49298419022a, current metadata version is 3|0||5499a69e00df49298419022a 2014-12-23T17:30:13.268+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-46 from shard version 3|0||5499a69e00df49298419022a to shard version 3|1||5499a69e00df49298419022a 2014-12-23T17:30:13.268+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a69e00df49298419022a, took 0ms 2014-12-23T17:30:13.268+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a69e00df49298419022a 2014-12-23T17:30:13.572+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:13-5499a6a59a8f659f34bfb76d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355813572), what: "split", ns: "benchdb1.COL-46", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a69e00df49298419022a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a69e00df49298419022a') } } } 2014-12-23T17:30:13.954+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-46/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:13.954+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-46", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-46-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a69e00df49298419022a') } ntoreturn:1 keyUpdates:0 reslen:37 1294ms 2014-12-23T17:30:16.278+0000 I INDEX [conn2] build index on: benchdb1.COL-47 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-47" } 2014-12-23T17:30:16.278+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:16.279+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:16.599+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-47 with requested shard version 1|2||5499a6a500df49298419022e, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:16.600+0000 I SHARDING [conn2] collection benchdb1.COL-47 was previously unsharded, new metadata loaded with shard version 1|2||5499a6a500df49298419022e 2014-12-23T17:30:16.600+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6a500df49298419022e, took 0ms 2014-12-23T17:30:16.644+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:16.644+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6a500df49298419022e') } 2014-12-23T17:30:16.963+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6a89a8f659f34bfb76f 2014-12-23T17:30:16.963+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:16-5499a6a89a8f659f34bfb770", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355816963), what: "moveChunk.start", ns: "benchdb1.COL-47", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:17.021+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-47 based on current shard version 1|2||5499a6a500df49298419022e, current metadata version is 1|2||5499a6a500df49298419022e 2014-12-23T17:30:17.022+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-47 already up to date (shard version : 1|2||5499a6a500df49298419022e, took 0ms) 2014-12-23T17:30:17.022+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6a500df49298419022e 2014-12-23T17:30:17.023+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:17.025+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.028+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.032+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.032+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:17.032+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:17.032+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6a500df49298419022e 2014-12-23T17:30:17.086+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:17.086+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6a500df49298419022e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-47' 2014-12-23T17:30:17.279+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:17-5499a6a99a8f659f34bfb771", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355817279), what: "moveChunk.commit", ns: "benchdb1.COL-47", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:17.309+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:17.309+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:17.309+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-47 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3944 2014-12-23T17:30:17.309+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-47 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:17.309+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:17.487+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:17.487+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:17-5499a6a99a8f659f34bfb772", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355817487), what: "moveChunk.from", ns: "benchdb1.COL-47", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 378, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 277, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:17.536+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6a500df49298419022e') } ntoreturn:1 keyUpdates:0 reslen:37 891ms 2014-12-23T17:30:17.537+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:17.537+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6a500df49298419022e') } 2014-12-23T17:30:17.855+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6a99a8f659f34bfb773 2014-12-23T17:30:17.855+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:17-5499a6a99a8f659f34bfb774", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355817855), what: "moveChunk.start", ns: "benchdb1.COL-47", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:17.908+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-47 based on current shard version 2|0||5499a6a500df49298419022e, current metadata version is 2|0||5499a6a500df49298419022e 2014-12-23T17:30:17.909+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-47 from shard version 2|0||5499a6a500df49298419022e to shard version 2|1||5499a6a500df49298419022e 2014-12-23T17:30:17.909+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6a500df49298419022e, took 0ms 2014-12-23T17:30:17.909+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6a500df49298419022e 2014-12-23T17:30:17.910+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:17.912+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.914+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.919+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.927+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.944+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:17.944+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:17.944+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:17.944+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6a500df49298419022e 2014-12-23T17:30:17.997+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:17.997+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6a500df49298419022e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-47' 2014-12-23T17:30:18.207+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:18-5499a6aa9a8f659f34bfb775", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355818207), what: "moveChunk.commit", ns: "benchdb1.COL-47", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:18.240+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:18.240+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:18.240+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-47 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3952 2014-12-23T17:30:18.240+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-47 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:18.240+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:18.412+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:18.413+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:18-5499a6aa9a8f659f34bfb776", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355818412), what: "moveChunk.from", ns: "benchdb1.COL-47", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 372, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 296, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:18.474+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-47", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-47-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6a500df49298419022e') } ntoreturn:1 keyUpdates:0 reslen:37 936ms 2014-12-23T17:30:18.475+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-47", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-47-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6a500df49298419022e') } 2014-12-23T17:30:19.013+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6aa9a8f659f34bfb777 2014-12-23T17:30:19.013+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-47 based on current shard version 3|0||5499a6a500df49298419022e, current metadata version is 3|0||5499a6a500df49298419022e 2014-12-23T17:30:19.014+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-47 from shard version 3|0||5499a6a500df49298419022e to shard version 3|1||5499a6a500df49298419022e 2014-12-23T17:30:19.014+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6a500df49298419022e, took 0ms 2014-12-23T17:30:19.014+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6a500df49298419022e 2014-12-23T17:30:19.318+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:19-5499a6ab9a8f659f34bfb778", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355819318), what: "split", ns: "benchdb1.COL-47", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6a500df49298419022e') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6a500df49298419022e') } } } 2014-12-23T17:30:19.706+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-47/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:19.706+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-47", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-47-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6a500df49298419022e') } ntoreturn:1 keyUpdates:0 reslen:37 1231ms 2014-12-23T17:30:22.308+0000 I INDEX [conn2] build index on: benchdb1.COL-48 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-48" } 2014-12-23T17:30:22.308+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:22.308+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:22.627+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-48 with requested shard version 1|2||5499a6ab00df492984190232, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:22.628+0000 I SHARDING [conn2] collection benchdb1.COL-48 was previously unsharded, new metadata loaded with shard version 1|2||5499a6ab00df492984190232 2014-12-23T17:30:22.628+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6ab00df492984190232, took 0ms 2014-12-23T17:30:22.673+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:22.674+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6ab00df492984190232') } 2014-12-23T17:30:23.280+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6ae9a8f659f34bfb77a 2014-12-23T17:30:23.281+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:23-5499a6af9a8f659f34bfb77b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355823281), what: "moveChunk.start", ns: "benchdb1.COL-48", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:23.344+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-48 based on current shard version 1|2||5499a6ab00df492984190232, current metadata version is 1|2||5499a6ab00df492984190232 2014-12-23T17:30:23.345+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-48 already up to date (shard version : 1|2||5499a6ab00df492984190232, took 0ms) 2014-12-23T17:30:23.345+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6ab00df492984190232 2014-12-23T17:30:23.346+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:23.349+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:23.352+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:23.356+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:23.356+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:23.357+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:23.357+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6ab00df492984190232 2014-12-23T17:30:23.416+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:23.416+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6ab00df492984190232 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-48' 2014-12-23T17:30:23.721+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:23-5499a6af9a8f659f34bfb77c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355823721), what: "moveChunk.commit", ns: "benchdb1.COL-48", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:23.787+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:23.787+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:23.787+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-48 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 3971 2014-12-23T17:30:23.787+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-48 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:23.787+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:24.090+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:24.090+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:24-5499a6b09a8f659f34bfb77d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355824090), what: "moveChunk.from", ns: "benchdb1.COL-48", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 430, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:24.147+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6ab00df492984190232') } ntoreturn:1 keyUpdates:0 reslen:37 1473ms 2014-12-23T17:30:24.149+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:24.149+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6ab00df492984190232') } 2014-12-23T17:30:24.756+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6b09a8f659f34bfb77e 2014-12-23T17:30:24.756+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:24-5499a6b09a8f659f34bfb77f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355824756), what: "moveChunk.start", ns: "benchdb1.COL-48", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:24.820+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-48 based on current shard version 2|0||5499a6ab00df492984190232, current metadata version is 2|0||5499a6ab00df492984190232 2014-12-23T17:30:24.821+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-48 from shard version 2|0||5499a6ab00df492984190232 to shard version 2|1||5499a6ab00df492984190232 2014-12-23T17:30:24.821+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6ab00df492984190232, took 0ms 2014-12-23T17:30:24.821+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6ab00df492984190232 2014-12-23T17:30:24.821+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:24.824+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:24.826+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:24.830+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:24.839+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:24.855+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:24.855+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:24.856+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:24.856+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6ab00df492984190232 2014-12-23T17:30:24.910+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:24.910+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6ab00df492984190232 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-48' 2014-12-23T17:30:25.214+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:25-5499a6b19a8f659f34bfb780", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355825214), what: "moveChunk.commit", ns: "benchdb1.COL-48", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:25.270+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:25.270+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:25.270+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-48 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 3979 2014-12-23T17:30:25.271+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-48 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:25.271+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:25.574+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:25.574+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:25-5499a6b19a8f659f34bfb781", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355825574), what: "moveChunk.from", ns: "benchdb1.COL-48", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 415, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:25.632+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-48", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-48-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6ab00df492984190232') } ntoreturn:1 keyUpdates:0 reslen:37 1482ms 2014-12-23T17:30:25.634+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-48", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-48-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6ab00df492984190232') } 2014-12-23T17:30:26.240+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6b19a8f659f34bfb782 2014-12-23T17:30:26.240+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-48 based on current shard version 3|0||5499a6ab00df492984190232, current metadata version is 3|0||5499a6ab00df492984190232 2014-12-23T17:30:26.241+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-48 from shard version 3|0||5499a6ab00df492984190232 to shard version 3|1||5499a6ab00df492984190232 2014-12-23T17:30:26.241+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6ab00df492984190232, took 0ms 2014-12-23T17:30:26.241+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6ab00df492984190232 2014-12-23T17:30:26.546+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:26-5499a6b29a8f659f34bfb783", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355826546), what: "split", ns: "benchdb1.COL-48", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6ab00df492984190232') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6ab00df492984190232') } } } 2014-12-23T17:30:26.923+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-48/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:26.923+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-48", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-48-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6ab00df492984190232') } ntoreturn:1 keyUpdates:0 reslen:37 1289ms 2014-12-23T17:30:29.376+0000 I INDEX [conn2] build index on: benchdb1.COL-49 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-49" } 2014-12-23T17:30:29.376+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:29.377+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:29.648+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-49 with requested shard version 1|2||5499a6b200df492984190236, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:29.649+0000 I SHARDING [conn2] collection benchdb1.COL-49 was previously unsharded, new metadata loaded with shard version 1|2||5499a6b200df492984190236 2014-12-23T17:30:29.649+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6b200df492984190236, took 0ms 2014-12-23T17:30:29.690+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:29.690+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b200df492984190236') } 2014-12-23T17:30:29.980+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6b59a8f659f34bfb785 2014-12-23T17:30:29.980+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:29-5499a6b59a8f659f34bfb786", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355829980), what: "moveChunk.start", ns: "benchdb1.COL-49", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:30.084+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-49 based on current shard version 1|2||5499a6b200df492984190236, current metadata version is 1|2||5499a6b200df492984190236 2014-12-23T17:30:30.085+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-49 already up to date (shard version : 1|2||5499a6b200df492984190236, took 0ms) 2014-12-23T17:30:30.085+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6b200df492984190236 2014-12-23T17:30:30.085+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:30.088+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.090+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.094+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.094+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:30.095+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:30.095+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6b200df492984190236 2014-12-23T17:30:30.144+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:30.144+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6b200df492984190236 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-49' 2014-12-23T17:30:30.321+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:30-5499a6b69a8f659f34bfb787", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355830321), what: "moveChunk.commit", ns: "benchdb1.COL-49", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:30.378+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:30.378+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:30.378+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-49 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4000 2014-12-23T17:30:30.378+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-49 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:30.378+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:30.552+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:30.552+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:30-5499a6b69a8f659f34bfb788", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355830552), what: "moveChunk.from", ns: "benchdb1.COL-49", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 394, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 283, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:30.612+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b200df492984190236') } ntoreturn:1 keyUpdates:0 reslen:37 921ms 2014-12-23T17:30:30.613+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:30.613+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b200df492984190236') } 2014-12-23T17:30:30.906+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6b69a8f659f34bfb789 2014-12-23T17:30:30.906+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:30-5499a6b69a8f659f34bfb78a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355830906), what: "moveChunk.start", ns: "benchdb1.COL-49", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:30.962+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-49 based on current shard version 2|0||5499a6b200df492984190236, current metadata version is 2|0||5499a6b200df492984190236 2014-12-23T17:30:30.962+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-49 from shard version 2|0||5499a6b200df492984190236 to shard version 2|1||5499a6b200df492984190236 2014-12-23T17:30:30.962+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6b200df492984190236, took 0ms 2014-12-23T17:30:30.962+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6b200df492984190236 2014-12-23T17:30:30.963+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:30.965+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.967+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.972+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.980+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.997+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:30.997+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:30.997+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:30.997+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6b200df492984190236 2014-12-23T17:30:31.055+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:31.055+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6b200df492984190236 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-49' 2014-12-23T17:30:31.251+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:31-5499a6b79a8f659f34bfb78b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355831251), what: "moveChunk.commit", ns: "benchdb1.COL-49", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:31.305+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:31.305+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:31.305+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-49 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4008 2014-12-23T17:30:31.305+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-49 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:31.305+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:31.470+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:31.470+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:31-5499a6b79a8f659f34bfb78c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355831470), what: "moveChunk.from", ns: "benchdb1.COL-49", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 348, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 307, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:31.529+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-49", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-49-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b200df492984190236') } ntoreturn:1 keyUpdates:0 reslen:37 916ms 2014-12-23T17:30:31.531+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-49", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-49-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6b200df492984190236') } 2014-12-23T17:30:31.942+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6b79a8f659f34bfb78d 2014-12-23T17:30:31.942+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-49 based on current shard version 3|0||5499a6b200df492984190236, current metadata version is 3|0||5499a6b200df492984190236 2014-12-23T17:30:31.943+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-49 from shard version 3|0||5499a6b200df492984190236 to shard version 3|1||5499a6b200df492984190236 2014-12-23T17:30:31.943+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6b200df492984190236, took 0ms 2014-12-23T17:30:31.943+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6b200df492984190236 2014-12-23T17:30:32.248+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:32-5499a6b89a8f659f34bfb78e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355832248), what: "split", ns: "benchdb1.COL-49", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6b200df492984190236') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6b200df492984190236') } } } 2014-12-23T17:30:32.631+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-49/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:32.631+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-49", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-49-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6b200df492984190236') } ntoreturn:1 keyUpdates:0 reslen:37 1100ms 2014-12-23T17:30:35.225+0000 I INDEX [conn2] build index on: benchdb1.COL-50 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-50" } 2014-12-23T17:30:35.225+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:35.226+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:35.540+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-50 with requested shard version 1|2||5499a6b800df49298419023a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:35.540+0000 I SHARDING [conn2] collection benchdb1.COL-50 was previously unsharded, new metadata loaded with shard version 1|2||5499a6b800df49298419023a 2014-12-23T17:30:35.540+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6b800df49298419023a, took 0ms 2014-12-23T17:30:35.586+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:35.586+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b800df49298419023a') } 2014-12-23T17:30:36.194+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6bb9a8f659f34bfb790 2014-12-23T17:30:36.194+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:36-5499a6bc9a8f659f34bfb791", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355836194), what: "moveChunk.start", ns: "benchdb1.COL-50", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:36.258+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-50 based on current shard version 1|2||5499a6b800df49298419023a, current metadata version is 1|2||5499a6b800df49298419023a 2014-12-23T17:30:36.259+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-50 already up to date (shard version : 1|2||5499a6b800df49298419023a, took 0ms) 2014-12-23T17:30:36.259+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6b800df49298419023a 2014-12-23T17:30:36.260+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:36.262+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:36.265+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:36.269+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:36.269+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:36.270+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:36.270+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6b800df49298419023a 2014-12-23T17:30:36.331+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:36.331+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6b800df49298419023a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-50' 2014-12-23T17:30:36.635+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:36-5499a6bc9a8f659f34bfb792", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355836635), what: "moveChunk.commit", ns: "benchdb1.COL-50", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:36.738+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:36.738+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:36.738+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-50 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4027 2014-12-23T17:30:36.738+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-50 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:36.738+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:37.041+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:37.041+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:37-5499a6bd9a8f659f34bfb793", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355837041), what: "moveChunk.from", ns: "benchdb1.COL-50", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 468, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:37.097+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b800df49298419023a') } ntoreturn:1 keyUpdates:0 reslen:37 1511ms 2014-12-23T17:30:37.099+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:37.099+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b800df49298419023a') } 2014-12-23T17:30:37.706+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6bd9a8f659f34bfb794 2014-12-23T17:30:37.706+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:37-5499a6bd9a8f659f34bfb795", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355837706), what: "moveChunk.start", ns: "benchdb1.COL-50", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:37.770+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-50 based on current shard version 2|0||5499a6b800df49298419023a, current metadata version is 2|0||5499a6b800df49298419023a 2014-12-23T17:30:37.771+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-50 from shard version 2|0||5499a6b800df49298419023a to shard version 2|1||5499a6b800df49298419023a 2014-12-23T17:30:37.771+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6b800df49298419023a, took 0ms 2014-12-23T17:30:37.771+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6b800df49298419023a 2014-12-23T17:30:37.771+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:37.774+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:37.776+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:37.781+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:37.789+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:37.805+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:37.805+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:37.806+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:37.806+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6b800df49298419023a 2014-12-23T17:30:37.863+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:37.863+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6b800df49298419023a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-50' 2014-12-23T17:30:38.167+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:38-5499a6be9a8f659f34bfb796", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355838167), what: "moveChunk.commit", ns: "benchdb1.COL-50", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:38.226+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:38.226+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:38.226+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-50 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4035 2014-12-23T17:30:38.226+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-50 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:38.226+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:38.529+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:38.529+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:38-5499a6be9a8f659f34bfb797", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355838529), what: "moveChunk.from", ns: "benchdb1.COL-50", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 420, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:38.588+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-50", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-50-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6b800df49298419023a') } ntoreturn:1 keyUpdates:0 reslen:37 1489ms 2014-12-23T17:30:38.590+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-50", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-50-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6b800df49298419023a') } 2014-12-23T17:30:39.198+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6be9a8f659f34bfb798 2014-12-23T17:30:39.198+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-50 based on current shard version 3|0||5499a6b800df49298419023a, current metadata version is 3|0||5499a6b800df49298419023a 2014-12-23T17:30:39.199+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-50 from shard version 3|0||5499a6b800df49298419023a to shard version 3|1||5499a6b800df49298419023a 2014-12-23T17:30:39.199+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6b800df49298419023a, took 0ms 2014-12-23T17:30:39.199+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6b800df49298419023a 2014-12-23T17:30:39.503+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:39-5499a6bf9a8f659f34bfb799", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355839503), what: "split", ns: "benchdb1.COL-50", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6b800df49298419023a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6b800df49298419023a') } } } 2014-12-23T17:30:39.835+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-50/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:39.835+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-50", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-50-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6b800df49298419023a') } ntoreturn:1 keyUpdates:0 reslen:37 1244ms 2014-12-23T17:30:40.190+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:30:39 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:30:42.344+0000 I INDEX [conn2] build index on: benchdb1.COL-51 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-51" } 2014-12-23T17:30:42.344+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:42.344+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:42.697+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-51 with requested shard version 1|2||5499a6bf00df49298419023e, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:42.698+0000 I SHARDING [conn2] collection benchdb1.COL-51 was previously unsharded, new metadata loaded with shard version 1|2||5499a6bf00df49298419023e 2014-12-23T17:30:42.698+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6bf00df49298419023e, took 0ms 2014-12-23T17:30:42.749+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:42.749+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6bf00df49298419023e') } 2014-12-23T17:30:43.072+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6c29a8f659f34bfb79b 2014-12-23T17:30:43.072+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:43-5499a6c39a8f659f34bfb79c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355843072), what: "moveChunk.start", ns: "benchdb1.COL-51", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:43.121+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-51 based on current shard version 1|2||5499a6bf00df49298419023e, current metadata version is 1|2||5499a6bf00df49298419023e 2014-12-23T17:30:43.122+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-51 already up to date (shard version : 1|2||5499a6bf00df49298419023e, took 0ms) 2014-12-23T17:30:43.122+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6bf00df49298419023e 2014-12-23T17:30:43.122+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:43.125+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:43.127+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:43.132+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:43.132+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:43.132+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:43.132+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6bf00df49298419023e 2014-12-23T17:30:43.199+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:43.199+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6bf00df49298419023e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-51' 2014-12-23T17:30:43.383+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:43-5499a6c39a8f659f34bfb79d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355843383), what: "moveChunk.commit", ns: "benchdb1.COL-51", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:43.439+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:43.439+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:43.439+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-51 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4056 2014-12-23T17:30:43.439+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-51 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:43.439+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:43.614+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:43.614+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:43-5499a6c39a8f659f34bfb79e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355843614), what: "moveChunk.from", ns: "benchdb1.COL-51", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 372, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 307, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:43.665+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6bf00df49298419023e') } ntoreturn:1 keyUpdates:0 reslen:37 916ms 2014-12-23T17:30:43.667+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:43.667+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6bf00df49298419023e') } 2014-12-23T17:30:43.966+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6c39a8f659f34bfb79f 2014-12-23T17:30:43.966+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:43-5499a6c39a8f659f34bfb7a0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355843966), what: "moveChunk.start", ns: "benchdb1.COL-51", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:44.025+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-51 based on current shard version 2|0||5499a6bf00df49298419023e, current metadata version is 2|0||5499a6bf00df49298419023e 2014-12-23T17:30:44.026+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-51 from shard version 2|0||5499a6bf00df49298419023e to shard version 2|1||5499a6bf00df49298419023e 2014-12-23T17:30:44.026+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6bf00df49298419023e, took 0ms 2014-12-23T17:30:44.026+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6bf00df49298419023e 2014-12-23T17:30:44.026+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:44.028+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:44.031+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:44.035+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:44.043+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:44.060+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:44.060+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:44.060+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:44.060+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6bf00df49298419023e 2014-12-23T17:30:44.099+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:44.099+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6bf00df49298419023e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-51' 2014-12-23T17:30:44.284+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:44-5499a6c49a8f659f34bfb7a1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355844284), what: "moveChunk.commit", ns: "benchdb1.COL-51", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:44.346+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:44.346+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:44.346+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-51 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4064 2014-12-23T17:30:44.346+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-51 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:44.346+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:44.501+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:44.501+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:44-5499a6c49a8f659f34bfb7a2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355844501), what: "moveChunk.from", ns: "benchdb1.COL-51", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 358, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 285, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:44.556+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-51", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-51-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6bf00df49298419023e') } ntoreturn:1 keyUpdates:0 reslen:37 889ms 2014-12-23T17:30:44.558+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-51", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-51-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6bf00df49298419023e') } 2014-12-23T17:30:44.827+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6c49a8f659f34bfb7a3 2014-12-23T17:30:44.827+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-51 based on current shard version 3|0||5499a6bf00df49298419023e, current metadata version is 3|0||5499a6bf00df49298419023e 2014-12-23T17:30:44.828+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-51 from shard version 3|0||5499a6bf00df49298419023e to shard version 3|1||5499a6bf00df49298419023e 2014-12-23T17:30:44.828+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6bf00df49298419023e, took 0ms 2014-12-23T17:30:44.828+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6bf00df49298419023e 2014-12-23T17:30:45.091+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:45-5499a6c59a8f659f34bfb7a4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355845091), what: "split", ns: "benchdb1.COL-51", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6bf00df49298419023e') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6bf00df49298419023e') } } } 2014-12-23T17:30:45.475+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-51/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:45.475+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-51", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-51-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6bf00df49298419023e') } ntoreturn:1 keyUpdates:0 reslen:37 916ms 2014-12-23T17:30:47.992+0000 I INDEX [conn2] build index on: benchdb1.COL-52 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-52" } 2014-12-23T17:30:47.992+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:47.993+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:48.282+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-52 with requested shard version 1|2||5499a6c400df492984190242, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:48.282+0000 I SHARDING [conn2] collection benchdb1.COL-52 was previously unsharded, new metadata loaded with shard version 1|2||5499a6c400df492984190242 2014-12-23T17:30:48.282+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6c400df492984190242, took 0ms 2014-12-23T17:30:48.322+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:48.322+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6c400df492984190242') } 2014-12-23T17:30:48.929+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6c89a8f659f34bfb7a6 2014-12-23T17:30:48.929+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:48-5499a6c89a8f659f34bfb7a7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355848929), what: "moveChunk.start", ns: "benchdb1.COL-52", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:48.993+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 based on current shard version 1|2||5499a6c400df492984190242, current metadata version is 1|2||5499a6c400df492984190242 2014-12-23T17:30:48.994+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-52 already up to date (shard version : 1|2||5499a6c400df492984190242, took 0ms) 2014-12-23T17:30:48.994+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6c400df492984190242 2014-12-23T17:30:48.994+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:48.997+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:48.999+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:49.004+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:49.004+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:49.004+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:49.004+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6c400df492984190242 2014-12-23T17:30:49.058+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:49.058+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6c400df492984190242 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-52' 2014-12-23T17:30:49.363+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:49-5499a6c99a8f659f34bfb7a8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355849363), what: "moveChunk.commit", ns: "benchdb1.COL-52", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:49.424+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:49.424+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:49.424+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-52 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4082 2014-12-23T17:30:49.424+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-52 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:49.424+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:49.728+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:49.728+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:49-5499a6c99a8f659f34bfb7a9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355849728), what: "moveChunk.from", ns: "benchdb1.COL-52", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 420, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:49.783+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6c400df492984190242') } ntoreturn:1 keyUpdates:0 reslen:37 1460ms 2014-12-23T17:30:49.784+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:49.784+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6c400df492984190242') } 2014-12-23T17:30:50.392+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6c99a8f659f34bfb7aa 2014-12-23T17:30:50.392+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:50-5499a6ca9a8f659f34bfb7ab", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355850392), what: "moveChunk.start", ns: "benchdb1.COL-52", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:50.454+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 based on current shard version 2|0||5499a6c400df492984190242, current metadata version is 2|0||5499a6c400df492984190242 2014-12-23T17:30:50.455+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-52 from shard version 2|0||5499a6c400df492984190242 to shard version 2|1||5499a6c400df492984190242 2014-12-23T17:30:50.455+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6c400df492984190242, took 0ms 2014-12-23T17:30:50.455+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6c400df492984190242 2014-12-23T17:30:50.456+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:50.458+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:50.460+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:50.465+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:50.473+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:50.490+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:50.490+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:50.490+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:50.490+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6c400df492984190242 2014-12-23T17:30:50.550+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:50.550+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6c400df492984190242 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-52' 2014-12-23T17:30:50.855+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:50-5499a6ca9a8f659f34bfb7ac", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355850855), what: "moveChunk.commit", ns: "benchdb1.COL-52", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:50.913+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:50.913+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:50.913+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-52 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4090 2014-12-23T17:30:50.913+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-52 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:50.913+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:51.217+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:51.217+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:51-5499a6cb9a8f659f34bfb7ad", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355851217), what: "moveChunk.from", ns: "benchdb1.COL-52", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:51.275+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-52", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-52-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6c400df492984190242') } ntoreturn:1 keyUpdates:0 reslen:37 1491ms 2014-12-23T17:30:51.277+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-52", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-52-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6c400df492984190242') } 2014-12-23T17:30:51.884+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6cb9a8f659f34bfb7ae 2014-12-23T17:30:51.884+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 based on current shard version 3|0||5499a6c400df492984190242, current metadata version is 3|0||5499a6c400df492984190242 2014-12-23T17:30:51.885+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-52 from shard version 3|0||5499a6c400df492984190242 to shard version 3|1||5499a6c400df492984190242 2014-12-23T17:30:51.885+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6c400df492984190242, took 0ms 2014-12-23T17:30:51.885+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6c400df492984190242 2014-12-23T17:30:52.190+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:52-5499a6cc9a8f659f34bfb7af", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355852190), what: "split", ns: "benchdb1.COL-52", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6c400df492984190242') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6c400df492984190242') } } } 2014-12-23T17:30:52.572+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-52/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:52.572+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-52", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-52-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6c400df492984190242') } ntoreturn:1 keyUpdates:0 reslen:37 1294ms 2014-12-23T17:30:55.154+0000 I INDEX [conn2] build index on: benchdb1.COL-53 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-53" } 2014-12-23T17:30:55.154+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:30:55.155+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:30:55.582+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-53 with requested shard version 1|2||5499a6cc00df492984190246, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:30:55.583+0000 I SHARDING [conn2] collection benchdb1.COL-53 was previously unsharded, new metadata loaded with shard version 1|2||5499a6cc00df492984190246 2014-12-23T17:30:55.583+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6cc00df492984190246, took 0ms 2014-12-23T17:30:55.644+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:55.645+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6cc00df492984190246') } 2014-12-23T17:30:55.922+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6cf9a8f659f34bfb7b1 2014-12-23T17:30:55.922+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:55-5499a6cf9a8f659f34bfb7b2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355855922), what: "moveChunk.start", ns: "benchdb1.COL-53", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:30:55.977+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-53 based on current shard version 1|2||5499a6cc00df492984190246, current metadata version is 1|2||5499a6cc00df492984190246 2014-12-23T17:30:55.978+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-53 already up to date (shard version : 1|2||5499a6cc00df492984190246, took 0ms) 2014-12-23T17:30:55.978+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6cc00df492984190246 2014-12-23T17:30:55.978+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:55.981+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:55.983+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:55.988+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:55.988+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:55.988+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:55.988+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6cc00df492984190246 2014-12-23T17:30:56.036+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:56.036+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6cc00df492984190246 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-53' 2014-12-23T17:30:56.209+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:56-5499a6d09a8f659f34bfb7b3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355856209), what: "moveChunk.commit", ns: "benchdb1.COL-53", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:56.261+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:56.261+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:56.261+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-53 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4111 2014-12-23T17:30:56.262+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-53 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:30:56.262+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:56.438+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:56.438+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:56-5499a6d09a8f659f34bfb7b4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355856438), what: "moveChunk.from", ns: "benchdb1.COL-53", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 333, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 273, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:30:56.491+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6cc00df492984190246') } ntoreturn:1 keyUpdates:0 reslen:37 847ms 2014-12-23T17:30:56.493+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:30:56.493+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6cc00df492984190246') } 2014-12-23T17:30:56.778+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6d09a8f659f34bfb7b5 2014-12-23T17:30:56.778+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:56-5499a6d09a8f659f34bfb7b6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355856778), what: "moveChunk.start", ns: "benchdb1.COL-53", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:30:56.834+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-53 based on current shard version 2|0||5499a6cc00df492984190246, current metadata version is 2|0||5499a6cc00df492984190246 2014-12-23T17:30:56.835+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-53 from shard version 2|0||5499a6cc00df492984190246 to shard version 2|1||5499a6cc00df492984190246 2014-12-23T17:30:56.835+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6cc00df492984190246, took 0ms 2014-12-23T17:30:56.835+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6cc00df492984190246 2014-12-23T17:30:56.835+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:30:56.838+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:56.840+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:56.844+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:56.853+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:56.869+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:30:56.869+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:30:56.870+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:30:56.870+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6cc00df492984190246 2014-12-23T17:30:56.922+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:30:56.922+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6cc00df492984190246 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-53' 2014-12-23T17:30:57.116+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:57-5499a6d19a8f659f34bfb7b7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355857116), what: "moveChunk.commit", ns: "benchdb1.COL-53", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:30:57.167+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:57.167+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:30:57.168+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-53 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4119 2014-12-23T17:30:57.168+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-53 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:30:57.168+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:30:57.331+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:57.331+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:57-5499a6d19a8f659f34bfb7b8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355857331), what: "moveChunk.from", ns: "benchdb1.COL-53", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 341, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 298, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:30:57.387+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-53", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-53-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6cc00df492984190246') } ntoreturn:1 keyUpdates:0 reslen:37 894ms 2014-12-23T17:30:57.389+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-53", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-53-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6cc00df492984190246') } 2014-12-23T17:30:57.693+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6d19a8f659f34bfb7b9 2014-12-23T17:30:57.693+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-53 based on current shard version 3|0||5499a6cc00df492984190246, current metadata version is 3|0||5499a6cc00df492984190246 2014-12-23T17:30:57.693+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-53 from shard version 3|0||5499a6cc00df492984190246 to shard version 3|1||5499a6cc00df492984190246 2014-12-23T17:30:57.693+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6cc00df492984190246, took 0ms 2014-12-23T17:30:57.694+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6cc00df492984190246 2014-12-23T17:30:57.999+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:30:57-5499a6d19a8f659f34bfb7ba", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355857999), what: "split", ns: "benchdb1.COL-53", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6cc00df492984190246') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6cc00df492984190246') } } } 2014-12-23T17:30:58.381+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-53/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:30:58.381+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-53", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-53-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6cc00df492984190246') } ntoreturn:1 keyUpdates:0 reslen:37 991ms 2014-12-23T17:31:00.989+0000 I INDEX [conn2] build index on: benchdb1.COL-54 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-54" } 2014-12-23T17:31:00.989+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:00.990+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:01.320+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-54 with requested shard version 1|2||5499a6d100df49298419024a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:01.321+0000 I SHARDING [conn2] collection benchdb1.COL-54 was previously unsharded, new metadata loaded with shard version 1|2||5499a6d100df49298419024a 2014-12-23T17:31:01.321+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6d100df49298419024a, took 0ms 2014-12-23T17:31:01.363+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:01.364+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d100df49298419024a') } 2014-12-23T17:31:01.971+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6d59a8f659f34bfb7bc 2014-12-23T17:31:01.971+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:01-5499a6d59a8f659f34bfb7bd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355861971), what: "moveChunk.start", ns: "benchdb1.COL-54", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:02.039+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-54 based on current shard version 1|2||5499a6d100df49298419024a, current metadata version is 1|2||5499a6d100df49298419024a 2014-12-23T17:31:02.040+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-54 already up to date (shard version : 1|2||5499a6d100df49298419024a, took 0ms) 2014-12-23T17:31:02.040+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6d100df49298419024a 2014-12-23T17:31:02.040+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:02.043+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:02.045+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:02.050+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:02.050+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:02.050+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:02.050+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6d100df49298419024a 2014-12-23T17:31:02.109+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:02.110+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6d100df49298419024a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-54' 2014-12-23T17:31:02.415+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:02-5499a6d69a8f659f34bfb7be", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355862415), what: "moveChunk.commit", ns: "benchdb1.COL-54", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:02.478+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:02.478+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:02.478+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-54 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4137 2014-12-23T17:31:02.478+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-54 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:02.478+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:02.782+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:02.782+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:02-5499a6d69a8f659f34bfb7bf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355862782), what: "moveChunk.from", ns: "benchdb1.COL-54", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 675, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 428, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:02.846+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d100df49298419024a') } ntoreturn:1 keyUpdates:0 reslen:37 1482ms 2014-12-23T17:31:02.848+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:02.848+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d100df49298419024a') } 2014-12-23T17:31:03.429+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6d69a8f659f34bfb7c0 2014-12-23T17:31:03.429+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:03-5499a6d79a8f659f34bfb7c1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355863429), what: "moveChunk.start", ns: "benchdb1.COL-54", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:03.492+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-54 based on current shard version 2|0||5499a6d100df49298419024a, current metadata version is 2|0||5499a6d100df49298419024a 2014-12-23T17:31:03.493+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-54 from shard version 2|0||5499a6d100df49298419024a to shard version 2|1||5499a6d100df49298419024a 2014-12-23T17:31:03.493+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6d100df49298419024a, took 0ms 2014-12-23T17:31:03.493+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6d100df49298419024a 2014-12-23T17:31:03.493+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:03.495+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:03.498+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:03.502+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:03.511+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:03.527+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:03.527+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:03.528+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:03.528+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6d100df49298419024a 2014-12-23T17:31:03.584+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:03.584+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6d100df49298419024a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-54' 2014-12-23T17:31:03.889+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:03-5499a6d79a8f659f34bfb7c2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355863889), what: "moveChunk.commit", ns: "benchdb1.COL-54", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:03.949+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:03.949+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:03.949+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-54 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4145 2014-12-23T17:31:03.949+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-54 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:03.949+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:04.252+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:04.252+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:04-5499a6d89a8f659f34bfb7c3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355864252), what: "moveChunk.from", ns: "benchdb1.COL-54", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 644, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 421, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:04.311+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-54", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-54-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d100df49298419024a') } ntoreturn:1 keyUpdates:0 reslen:37 1463ms 2014-12-23T17:31:04.313+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-54", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-54-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6d100df49298419024a') } 2014-12-23T17:31:04.921+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6d89a8f659f34bfb7c4 2014-12-23T17:31:04.921+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-54 based on current shard version 3|0||5499a6d100df49298419024a, current metadata version is 3|0||5499a6d100df49298419024a 2014-12-23T17:31:04.921+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-54 from shard version 3|0||5499a6d100df49298419024a to shard version 3|1||5499a6d100df49298419024a 2014-12-23T17:31:04.922+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6d100df49298419024a, took 0ms 2014-12-23T17:31:04.922+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6d100df49298419024a 2014-12-23T17:31:05.233+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:05-5499a6d99a8f659f34bfb7c5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355865233), what: "split", ns: "benchdb1.COL-54", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6d100df49298419024a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6d100df49298419024a') } } } 2014-12-23T17:31:05.611+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-54/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:05.611+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-54", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-54-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6d100df49298419024a') } ntoreturn:1 keyUpdates:0 reslen:37 1298ms 2014-12-23T17:31:08.063+0000 I INDEX [conn2] build index on: benchdb1.COL-55 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-55" } 2014-12-23T17:31:08.063+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:08.063+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:08.405+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-55 with requested shard version 1|2||5499a6d900df49298419024e, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:08.406+0000 I SHARDING [conn2] collection benchdb1.COL-55 was previously unsharded, new metadata loaded with shard version 1|2||5499a6d900df49298419024e 2014-12-23T17:31:08.406+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6d900df49298419024e, took 0ms 2014-12-23T17:31:08.467+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:08.468+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d900df49298419024e') } 2014-12-23T17:31:08.759+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6dc9a8f659f34bfb7c7 2014-12-23T17:31:08.759+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:08-5499a6dc9a8f659f34bfb7c8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355868759), what: "moveChunk.start", ns: "benchdb1.COL-55", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:08.811+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-55 based on current shard version 1|2||5499a6d900df49298419024e, current metadata version is 1|2||5499a6d900df49298419024e 2014-12-23T17:31:08.812+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-55 already up to date (shard version : 1|2||5499a6d900df49298419024e, took 0ms) 2014-12-23T17:31:08.812+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6d900df49298419024e 2014-12-23T17:31:08.812+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:08.815+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:08.817+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:08.822+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:08.822+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:08.822+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:08.822+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6d900df49298419024e 2014-12-23T17:31:08.879+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:08.879+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6d900df49298419024e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-55' 2014-12-23T17:31:09.056+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:09-5499a6dd9a8f659f34bfb7c9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355869056), what: "moveChunk.commit", ns: "benchdb1.COL-55", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:09.103+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:09.103+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:09.103+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-55 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4166 2014-12-23T17:31:09.103+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-55 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:09.103+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:09.282+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:09.282+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:09-5499a6dd9a8f659f34bfb7ca", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355869282), what: "moveChunk.from", ns: "benchdb1.COL-55", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 343, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 281, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:09.337+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d900df49298419024e') } ntoreturn:1 keyUpdates:0 reslen:37 869ms 2014-12-23T17:31:09.338+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:09.339+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d900df49298419024e') } 2014-12-23T17:31:09.662+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6dd9a8f659f34bfb7cb 2014-12-23T17:31:09.662+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:09-5499a6dd9a8f659f34bfb7cc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355869662), what: "moveChunk.start", ns: "benchdb1.COL-55", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:09.729+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-55 based on current shard version 2|0||5499a6d900df49298419024e, current metadata version is 2|0||5499a6d900df49298419024e 2014-12-23T17:31:09.730+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-55 from shard version 2|0||5499a6d900df49298419024e to shard version 2|1||5499a6d900df49298419024e 2014-12-23T17:31:09.730+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6d900df49298419024e, took 0ms 2014-12-23T17:31:09.730+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6d900df49298419024e 2014-12-23T17:31:09.730+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:09.733+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:09.735+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:09.739+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:09.748+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:09.764+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:09.764+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:09.765+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:09.765+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6d900df49298419024e 2014-12-23T17:31:09.839+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:09.839+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6d900df49298419024e through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-55' 2014-12-23T17:31:10.046+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:10-5499a6de9a8f659f34bfb7cd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355870046), what: "moveChunk.commit", ns: "benchdb1.COL-55", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:10.094+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:10.094+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:10.094+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-55 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4174 2014-12-23T17:31:10.094+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-55 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:10.094+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:10.249+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:10.249+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:10-5499a6de9a8f659f34bfb7ce", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355870249), what: "moveChunk.from", ns: "benchdb1.COL-55", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 391, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 329, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:10.316+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-55", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-55-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6d900df49298419024e') } ntoreturn:1 keyUpdates:0 reslen:37 977ms 2014-12-23T17:31:10.318+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-55", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-55-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6d900df49298419024e') } 2014-12-23T17:31:10.409+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:31:10 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:31:10.579+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6de9a8f659f34bfb7cf 2014-12-23T17:31:10.579+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-55 based on current shard version 3|0||5499a6d900df49298419024e, current metadata version is 3|0||5499a6d900df49298419024e 2014-12-23T17:31:10.580+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-55 from shard version 3|0||5499a6d900df49298419024e to shard version 3|1||5499a6d900df49298419024e 2014-12-23T17:31:10.580+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6d900df49298419024e, took 0ms 2014-12-23T17:31:10.580+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6d900df49298419024e 2014-12-23T17:31:10.792+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:10-5499a6de9a8f659f34bfb7d0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355870792), what: "split", ns: "benchdb1.COL-55", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6d900df49298419024e') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6d900df49298419024e') } } } 2014-12-23T17:31:11.098+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-55/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:11.098+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-55", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-55-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6d900df49298419024e') } ntoreturn:1 keyUpdates:0 reslen:37 779ms 2014-12-23T17:31:13.646+0000 I INDEX [conn2] build index on: benchdb1.COL-56 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-56" } 2014-12-23T17:31:13.646+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:13.647+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:13.947+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-56 with requested shard version 1|2||5499a6de00df492984190252, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:13.948+0000 I SHARDING [conn2] collection benchdb1.COL-56 was previously unsharded, new metadata loaded with shard version 1|2||5499a6de00df492984190252 2014-12-23T17:31:13.948+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6de00df492984190252, took 0ms 2014-12-23T17:31:13.994+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:13.994+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6de00df492984190252') } 2014-12-23T17:31:14.602+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6e19a8f659f34bfb7d2 2014-12-23T17:31:14.602+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:14-5499a6e29a8f659f34bfb7d3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355874602), what: "moveChunk.start", ns: "benchdb1.COL-56", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:14.665+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-56 based on current shard version 1|2||5499a6de00df492984190252, current metadata version is 1|2||5499a6de00df492984190252 2014-12-23T17:31:14.666+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-56 already up to date (shard version : 1|2||5499a6de00df492984190252, took 0ms) 2014-12-23T17:31:14.666+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6de00df492984190252 2014-12-23T17:31:14.667+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:14.669+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:14.672+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:14.676+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:14.676+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:14.677+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:14.677+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6de00df492984190252 2014-12-23T17:31:14.741+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:14.741+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6de00df492984190252 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-56' 2014-12-23T17:31:15.046+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:15-5499a6e39a8f659f34bfb7d4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355875046), what: "moveChunk.commit", ns: "benchdb1.COL-56", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:15.104+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:15.104+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:15.104+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-56 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4193 2014-12-23T17:31:15.105+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-56 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:15.105+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:15.408+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:15.408+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:15-5499a6e39a8f659f34bfb7d5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355875408), what: "moveChunk.from", ns: "benchdb1.COL-56", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 428, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:15.465+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6de00df492984190252') } ntoreturn:1 keyUpdates:0 reslen:37 1470ms 2014-12-23T17:31:15.466+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:15.467+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6de00df492984190252') } 2014-12-23T17:31:16.075+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6e39a8f659f34bfb7d6 2014-12-23T17:31:16.075+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:16-5499a6e49a8f659f34bfb7d7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355876075), what: "moveChunk.start", ns: "benchdb1.COL-56", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:16.138+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-56 based on current shard version 2|0||5499a6de00df492984190252, current metadata version is 2|0||5499a6de00df492984190252 2014-12-23T17:31:16.139+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-56 from shard version 2|0||5499a6de00df492984190252 to shard version 2|1||5499a6de00df492984190252 2014-12-23T17:31:16.139+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6de00df492984190252, took 0ms 2014-12-23T17:31:16.139+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6de00df492984190252 2014-12-23T17:31:16.139+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:16.142+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:16.144+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:16.149+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:16.157+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:16.174+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:16.174+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:16.174+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:16.174+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6de00df492984190252 2014-12-23T17:31:16.231+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:16.231+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6de00df492984190252 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-56' 2014-12-23T17:31:16.537+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:16-5499a6e49a8f659f34bfb7d8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355876537), what: "moveChunk.commit", ns: "benchdb1.COL-56", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:16.635+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:16.635+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:16.635+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-56 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4201 2014-12-23T17:31:16.635+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-56 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:16.635+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:16.938+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:16.938+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:16-5499a6e49a8f659f34bfb7d9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355876938), what: "moveChunk.from", ns: "benchdb1.COL-56", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 461, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:16.999+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-56", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-56-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6de00df492984190252') } ntoreturn:1 keyUpdates:0 reslen:37 1532ms 2014-12-23T17:31:17.001+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-56", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-56-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6de00df492984190252') } 2014-12-23T17:31:17.562+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6e49a8f659f34bfb7da 2014-12-23T17:31:17.562+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-56 based on current shard version 3|0||5499a6de00df492984190252, current metadata version is 3|0||5499a6de00df492984190252 2014-12-23T17:31:17.563+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-56 from shard version 3|0||5499a6de00df492984190252 to shard version 3|1||5499a6de00df492984190252 2014-12-23T17:31:17.563+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6de00df492984190252, took 0ms 2014-12-23T17:31:17.563+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6de00df492984190252 2014-12-23T17:31:17.869+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:17-5499a6e59a8f659f34bfb7db", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355877869), what: "split", ns: "benchdb1.COL-56", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6de00df492984190252') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6de00df492984190252') } } } 2014-12-23T17:31:18.240+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-56/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:18.240+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-56", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-56-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6de00df492984190252') } ntoreturn:1 keyUpdates:0 reslen:37 1238ms 2014-12-23T17:31:20.824+0000 I INDEX [conn2] build index on: benchdb1.COL-57 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-57" } 2014-12-23T17:31:20.824+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:20.825+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:21.187+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-57 with requested shard version 1|2||5499a6e500df492984190255, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:21.188+0000 I SHARDING [conn2] collection benchdb1.COL-57 was previously unsharded, new metadata loaded with shard version 1|2||5499a6e500df492984190255 2014-12-23T17:31:21.188+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6e500df492984190255, took 0ms 2014-12-23T17:31:21.232+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:21.233+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6e500df492984190255') } 2014-12-23T17:31:21.644+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6e99a8f659f34bfb7dd 2014-12-23T17:31:21.644+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:21-5499a6e99a8f659f34bfb7de", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355881644), what: "moveChunk.start", ns: "benchdb1.COL-57", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:21.702+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-57 based on current shard version 1|2||5499a6e500df492984190255, current metadata version is 1|2||5499a6e500df492984190255 2014-12-23T17:31:21.703+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-57 already up to date (shard version : 1|2||5499a6e500df492984190255, took 0ms) 2014-12-23T17:31:21.703+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6e500df492984190255 2014-12-23T17:31:21.704+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:21.706+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:21.709+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:21.713+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:21.713+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:21.713+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:21.713+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6e500df492984190255 2014-12-23T17:31:21.760+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:21.760+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6e500df492984190255 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-57' 2014-12-23T17:31:21.968+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:21-5499a6e99a8f659f34bfb7df", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355881968), what: "moveChunk.commit", ns: "benchdb1.COL-57", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:22.015+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:22.015+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:22.016+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-57 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4221 2014-12-23T17:31:22.016+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-57 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:22.016+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:22.212+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:22.212+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:22-5499a6ea9a8f659f34bfb7e0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355882212), what: "moveChunk.from", ns: "benchdb1.COL-57", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 470, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 302, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:22.261+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6e500df492984190255') } ntoreturn:1 keyUpdates:0 reslen:37 1028ms 2014-12-23T17:31:22.263+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:22.263+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6e500df492984190255') } 2014-12-23T17:31:22.550+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6ea9a8f659f34bfb7e1 2014-12-23T17:31:22.550+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:22-5499a6ea9a8f659f34bfb7e2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355882550), what: "moveChunk.start", ns: "benchdb1.COL-57", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:22.617+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-57 based on current shard version 2|0||5499a6e500df492984190255, current metadata version is 2|0||5499a6e500df492984190255 2014-12-23T17:31:22.618+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-57 from shard version 2|0||5499a6e500df492984190255 to shard version 2|1||5499a6e500df492984190255 2014-12-23T17:31:22.618+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6e500df492984190255, took 0ms 2014-12-23T17:31:22.618+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6e500df492984190255 2014-12-23T17:31:22.618+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:22.621+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:22.623+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:22.627+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:22.636+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:22.653+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:22.653+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:22.653+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:22.653+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6e500df492984190255 2014-12-23T17:31:22.689+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:22.689+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6e500df492984190255 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-57' 2014-12-23T17:31:22.877+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:22-5499a6ea9a8f659f34bfb7e3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355882877), what: "moveChunk.commit", ns: "benchdb1.COL-57", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:22.935+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:22.935+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:22.935+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-57 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4230 2014-12-23T17:31:22.935+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-57 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:22.935+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:23.088+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:23.088+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:23-5499a6eb9a8f659f34bfb7e4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355883088), what: "moveChunk.from", ns: "benchdb1.COL-57", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 354, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 281, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:23.169+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-57", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-57-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6e500df492984190255') } ntoreturn:1 keyUpdates:0 reslen:37 906ms 2014-12-23T17:31:23.171+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-57", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-57-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6e500df492984190255') } 2014-12-23T17:31:23.503+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6eb9a8f659f34bfb7e5 2014-12-23T17:31:23.503+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-57 based on current shard version 3|0||5499a6e500df492984190255, current metadata version is 3|0||5499a6e500df492984190255 2014-12-23T17:31:23.504+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-57 from shard version 3|0||5499a6e500df492984190255 to shard version 3|1||5499a6e500df492984190255 2014-12-23T17:31:23.504+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6e500df492984190255, took 0ms 2014-12-23T17:31:23.504+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6e500df492984190255 2014-12-23T17:31:23.682+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:23-5499a6eb9a8f659f34bfb7e6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355883682), what: "split", ns: "benchdb1.COL-57", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6e500df492984190255') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6e500df492984190255') } } } 2014-12-23T17:31:23.914+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-57/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:23.915+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-57", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-57-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6e500df492984190255') } ntoreturn:1 keyUpdates:0 reslen:37 743ms 2014-12-23T17:31:26.432+0000 I INDEX [conn2] build index on: benchdb1.COL-58 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-58" } 2014-12-23T17:31:26.432+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:26.432+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:26.767+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-58 with requested shard version 1|2||5499a6eb00df49298419025a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:26.768+0000 I SHARDING [conn2] collection benchdb1.COL-58 was previously unsharded, new metadata loaded with shard version 1|2||5499a6eb00df49298419025a 2014-12-23T17:31:26.768+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6eb00df49298419025a, took 0ms 2014-12-23T17:31:26.811+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:26.811+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6eb00df49298419025a') } 2014-12-23T17:31:27.418+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6ee9a8f659f34bfb7e8 2014-12-23T17:31:27.418+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:27-5499a6ef9a8f659f34bfb7e9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355887418), what: "moveChunk.start", ns: "benchdb1.COL-58", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:27.488+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-58 based on current shard version 1|2||5499a6eb00df49298419025a, current metadata version is 1|2||5499a6eb00df49298419025a 2014-12-23T17:31:27.489+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-58 already up to date (shard version : 1|2||5499a6eb00df49298419025a, took 0ms) 2014-12-23T17:31:27.489+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6eb00df49298419025a 2014-12-23T17:31:27.490+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:27.493+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:27.495+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:27.499+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:27.499+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:27.500+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:27.500+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6eb00df49298419025a 2014-12-23T17:31:27.558+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:27.558+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6eb00df49298419025a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-58' 2014-12-23T17:31:27.863+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:27-5499a6ef9a8f659f34bfb7ea", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355887863), what: "moveChunk.commit", ns: "benchdb1.COL-58", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:27.920+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:27.920+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:27.920+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-58 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4249 2014-12-23T17:31:27.920+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-58 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:27.920+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:28.223+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:28.223+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:28-5499a6f09a8f659f34bfb7eb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355888223), what: "moveChunk.from", ns: "benchdb1.COL-58", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 677, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 420, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:28.277+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6eb00df49298419025a') } ntoreturn:1 keyUpdates:0 reslen:37 1465ms 2014-12-23T17:31:28.278+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:28.278+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6eb00df49298419025a') } 2014-12-23T17:31:28.885+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6f09a8f659f34bfb7ec 2014-12-23T17:31:28.885+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:28-5499a6f09a8f659f34bfb7ed", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355888885), what: "moveChunk.start", ns: "benchdb1.COL-58", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:28.941+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-58 based on current shard version 2|0||5499a6eb00df49298419025a, current metadata version is 2|0||5499a6eb00df49298419025a 2014-12-23T17:31:28.941+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-58 from shard version 2|0||5499a6eb00df49298419025a to shard version 2|1||5499a6eb00df49298419025a 2014-12-23T17:31:28.941+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6eb00df49298419025a, took 0ms 2014-12-23T17:31:28.941+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6eb00df49298419025a 2014-12-23T17:31:28.942+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:28.944+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:28.946+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:28.951+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:28.959+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:28.976+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:28.976+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:28.976+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:28.976+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6eb00df49298419025a 2014-12-23T17:31:29.031+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:29.031+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6eb00df49298419025a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-58' 2014-12-23T17:31:29.335+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:29-5499a6f19a8f659f34bfb7ee", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355889335), what: "moveChunk.commit", ns: "benchdb1.COL-58", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:29.399+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:29.399+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:29.399+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-58 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4257 2014-12-23T17:31:29.399+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-58 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:29.399+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:29.702+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:29.702+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:29-5499a6f19a8f659f34bfb7ef", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355889702), what: "moveChunk.from", ns: "benchdb1.COL-58", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 663, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:29.762+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-58", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-58-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6eb00df49298419025a') } ntoreturn:1 keyUpdates:0 reslen:37 1483ms 2014-12-23T17:31:29.764+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-58", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-58-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6eb00df49298419025a') } 2014-12-23T17:31:30.334+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6f19a8f659f34bfb7f0 2014-12-23T17:31:30.334+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-58 based on current shard version 3|0||5499a6eb00df49298419025a, current metadata version is 3|0||5499a6eb00df49298419025a 2014-12-23T17:31:30.335+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-58 from shard version 3|0||5499a6eb00df49298419025a to shard version 3|1||5499a6eb00df49298419025a 2014-12-23T17:31:30.335+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6eb00df49298419025a, took 0ms 2014-12-23T17:31:30.335+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6eb00df49298419025a 2014-12-23T17:31:30.639+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:30-5499a6f29a8f659f34bfb7f1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355890639), what: "split", ns: "benchdb1.COL-58", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6eb00df49298419025a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6eb00df49298419025a') } } } 2014-12-23T17:31:31.014+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-58/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:31.015+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-58", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-58-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6eb00df49298419025a') } ntoreturn:1 keyUpdates:0 reslen:37 1251ms 2014-12-23T17:31:33.589+0000 I INDEX [conn2] build index on: benchdb1.COL-59 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-59" } 2014-12-23T17:31:33.589+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:33.590+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:33.945+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-59 with requested shard version 1|2||5499a6f200df49298419025d, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:33.946+0000 I SHARDING [conn2] collection benchdb1.COL-59 was previously unsharded, new metadata loaded with shard version 1|2||5499a6f200df49298419025d 2014-12-23T17:31:33.946+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6f200df49298419025d, took 0ms 2014-12-23T17:31:33.997+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:33.998+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f200df49298419025d') } 2014-12-23T17:31:34.481+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6f59a8f659f34bfb7f3 2014-12-23T17:31:34.481+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:34-5499a6f69a8f659f34bfb7f4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355894481), what: "moveChunk.start", ns: "benchdb1.COL-59", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:34.554+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-59 based on current shard version 1|2||5499a6f200df49298419025d, current metadata version is 1|2||5499a6f200df49298419025d 2014-12-23T17:31:34.555+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-59 already up to date (shard version : 1|2||5499a6f200df49298419025d, took 0ms) 2014-12-23T17:31:34.555+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6f200df49298419025d 2014-12-23T17:31:34.555+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:34.558+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:34.560+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:34.564+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:34.564+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:34.565+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:34.565+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6f200df49298419025d 2014-12-23T17:31:34.640+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:34.640+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6f200df49298419025d through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-59' 2014-12-23T17:31:34.786+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:34-5499a6f69a8f659f34bfb7f5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355894786), what: "moveChunk.commit", ns: "benchdb1.COL-59", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:34.819+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:34.819+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:34.819+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-59 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4276 2014-12-23T17:31:34.819+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-59 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:34.819+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:35.004+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:35.004+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:35-5499a6f79a8f659f34bfb7f6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355895004), what: "moveChunk.from", ns: "benchdb1.COL-59", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 556, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 254, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:35.059+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f200df49298419025d') } ntoreturn:1 keyUpdates:0 reslen:37 1061ms 2014-12-23T17:31:35.060+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:35.061+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f200df49298419025d') } 2014-12-23T17:31:35.367+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6f79a8f659f34bfb7f7 2014-12-23T17:31:35.367+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:35-5499a6f79a8f659f34bfb7f8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355895367), what: "moveChunk.start", ns: "benchdb1.COL-59", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:35.429+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-59 based on current shard version 2|0||5499a6f200df49298419025d, current metadata version is 2|0||5499a6f200df49298419025d 2014-12-23T17:31:35.430+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-59 from shard version 2|0||5499a6f200df49298419025d to shard version 2|1||5499a6f200df49298419025d 2014-12-23T17:31:35.430+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6f200df49298419025d, took 0ms 2014-12-23T17:31:35.430+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6f200df49298419025d 2014-12-23T17:31:35.430+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:35.432+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:35.435+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:35.439+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:35.448+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:35.464+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:35.464+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:35.465+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:35.465+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6f200df49298419025d 2014-12-23T17:31:35.539+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:35.539+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6f200df49298419025d through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-59' 2014-12-23T17:31:35.733+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:35-5499a6f79a8f659f34bfb7f9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355895733), what: "moveChunk.commit", ns: "benchdb1.COL-59", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:35.790+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:35.790+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:35.790+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-59 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4286 2014-12-23T17:31:35.790+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-59 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:35.790+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:35.947+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:35.947+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:35-5499a6f79a8f659f34bfb7fa", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355895947), what: "moveChunk.from", ns: "benchdb1.COL-59", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 369, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 325, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:36.006+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-59", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-59-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f200df49298419025d') } ntoreturn:1 keyUpdates:0 reslen:37 945ms 2014-12-23T17:31:36.008+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-59", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-59-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6f200df49298419025d') } 2014-12-23T17:31:36.308+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6f89a8f659f34bfb7fb 2014-12-23T17:31:36.308+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-59 based on current shard version 3|0||5499a6f200df49298419025d, current metadata version is 3|0||5499a6f200df49298419025d 2014-12-23T17:31:36.309+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-59 from shard version 3|0||5499a6f200df49298419025d to shard version 3|1||5499a6f200df49298419025d 2014-12-23T17:31:36.309+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6f200df49298419025d, took 0ms 2014-12-23T17:31:36.309+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6f200df49298419025d 2014-12-23T17:31:36.519+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:36-5499a6f89a8f659f34bfb7fc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355896519), what: "split", ns: "benchdb1.COL-59", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6f200df49298419025d') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6f200df49298419025d') } } } 2014-12-23T17:31:36.730+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-59/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:36.731+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-59", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-59-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6f200df49298419025d') } ntoreturn:1 keyUpdates:0 reslen:37 722ms 2014-12-23T17:31:38.803+0000 I INDEX [conn2] build index on: benchdb1.COL-60 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-60" } 2014-12-23T17:31:38.803+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:38.804+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:39.123+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-60 with requested shard version 1|2||5499a6f700df492984190262, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:39.124+0000 I SHARDING [conn2] collection benchdb1.COL-60 was previously unsharded, new metadata loaded with shard version 1|2||5499a6f700df492984190262 2014-12-23T17:31:39.124+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6f700df492984190262, took 0ms 2014-12-23T17:31:39.165+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:39.166+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f700df492984190262') } 2014-12-23T17:31:39.772+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6fb9a8f659f34bfb7fe 2014-12-23T17:31:39.772+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:39-5499a6fb9a8f659f34bfb7ff", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355899772), what: "moveChunk.start", ns: "benchdb1.COL-60", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:39.838+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-60 based on current shard version 1|2||5499a6f700df492984190262, current metadata version is 1|2||5499a6f700df492984190262 2014-12-23T17:31:39.839+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-60 already up to date (shard version : 1|2||5499a6f700df492984190262, took 0ms) 2014-12-23T17:31:39.839+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6f700df492984190262 2014-12-23T17:31:39.840+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:39.843+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:39.845+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:39.849+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:39.849+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:39.850+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:39.850+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6f700df492984190262 2014-12-23T17:31:39.914+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:39.914+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6f700df492984190262 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-60' 2014-12-23T17:31:40.187+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:40-5499a6fc9a8f659f34bfb800", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355900187), what: "moveChunk.commit", ns: "benchdb1.COL-60", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:40.258+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:40.258+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:40.258+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-60 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4305 2014-12-23T17:31:40.258+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-60 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:40.258+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:40.515+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:40.515+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:40-5499a6fc9a8f659f34bfb801", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355900515), what: "moveChunk.from", ns: "benchdb1.COL-60", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 408, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:40.581+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f700df492984190262') } ntoreturn:1 keyUpdates:0 reslen:37 1415ms 2014-12-23T17:31:40.583+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:40.583+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f700df492984190262') } 2014-12-23T17:31:40.932+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:31:40 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:31:41.135+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6fc9a8f659f34bfb802 2014-12-23T17:31:41.135+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:41-5499a6fd9a8f659f34bfb803", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355901135), what: "moveChunk.start", ns: "benchdb1.COL-60", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:41.204+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-60 based on current shard version 2|0||5499a6f700df492984190262, current metadata version is 2|0||5499a6f700df492984190262 2014-12-23T17:31:41.205+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-60 from shard version 2|0||5499a6f700df492984190262 to shard version 2|1||5499a6f700df492984190262 2014-12-23T17:31:41.205+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6f700df492984190262, took 0ms 2014-12-23T17:31:41.205+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6f700df492984190262 2014-12-23T17:31:41.205+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:41.208+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:41.210+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:41.215+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:41.223+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:41.239+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:41.239+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:41.240+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:41.240+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6f700df492984190262 2014-12-23T17:31:41.296+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:41.296+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6f700df492984190262 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-60' 2014-12-23T17:31:41.601+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:41-5499a6fd9a8f659f34bfb804", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355901601), what: "moveChunk.commit", ns: "benchdb1.COL-60", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:41.664+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:41.664+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:41.664+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-60 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4313 2014-12-23T17:31:41.664+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-60 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:41.664+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:41.967+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:41.968+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:41-5499a6fd9a8f659f34bfb805", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355901968), what: "moveChunk.from", ns: "benchdb1.COL-60", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 622, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 424, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:42.026+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-60", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-60-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6f700df492984190262') } ntoreturn:1 keyUpdates:0 reslen:37 1443ms 2014-12-23T17:31:42.028+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-60", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-60-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6f700df492984190262') } 2014-12-23T17:31:42.635+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a6fe9a8f659f34bfb806 2014-12-23T17:31:42.635+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-60 based on current shard version 3|0||5499a6f700df492984190262, current metadata version is 3|0||5499a6f700df492984190262 2014-12-23T17:31:42.636+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-60 from shard version 3|0||5499a6f700df492984190262 to shard version 3|1||5499a6f700df492984190262 2014-12-23T17:31:42.636+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6f700df492984190262, took 0ms 2014-12-23T17:31:42.636+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6f700df492984190262 2014-12-23T17:31:42.943+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:42-5499a6fe9a8f659f34bfb807", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355902943), what: "split", ns: "benchdb1.COL-60", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6f700df492984190262') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6f700df492984190262') } } } 2014-12-23T17:31:43.330+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-60/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:43.330+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-60", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-60-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6f700df492984190262') } ntoreturn:1 keyUpdates:0 reslen:37 1302ms 2014-12-23T17:31:45.931+0000 I INDEX [conn2] build index on: benchdb1.COL-61 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-61" } 2014-12-23T17:31:45.931+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:45.932+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:46.241+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-61 with requested shard version 1|2||5499a6fe00df492984190265, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:46.242+0000 I SHARDING [conn2] collection benchdb1.COL-61 was previously unsharded, new metadata loaded with shard version 1|2||5499a6fe00df492984190265 2014-12-23T17:31:46.242+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a6fe00df492984190265, took 0ms 2014-12-23T17:31:46.283+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:46.284+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6fe00df492984190265') } 2014-12-23T17:31:46.841+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a7029a8f659f34bfb809 2014-12-23T17:31:46.841+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:46-5499a7029a8f659f34bfb80a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355906841), what: "moveChunk.start", ns: "benchdb1.COL-61", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:46.913+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-61 based on current shard version 1|2||5499a6fe00df492984190265, current metadata version is 1|2||5499a6fe00df492984190265 2014-12-23T17:31:46.914+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-61 already up to date (shard version : 1|2||5499a6fe00df492984190265, took 0ms) 2014-12-23T17:31:46.914+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a6fe00df492984190265 2014-12-23T17:31:46.915+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:46.917+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:46.920+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:46.924+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:46.924+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:46.924+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:46.924+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a6fe00df492984190265 2014-12-23T17:31:46.984+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:46.984+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a6fe00df492984190265 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-61' 2014-12-23T17:31:47.288+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:47-5499a7039a8f659f34bfb80b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355907288), what: "moveChunk.commit", ns: "benchdb1.COL-61", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:47.362+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:47.362+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:47.362+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-61 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4332 2014-12-23T17:31:47.362+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-61 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:47.362+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:47.607+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:47.607+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:47-5499a7039a8f659f34bfb80c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355907607), what: "moveChunk.from", ns: "benchdb1.COL-61", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 630, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 437, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:47.674+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6fe00df492984190265') } ntoreturn:1 keyUpdates:0 reslen:37 1390ms 2014-12-23T17:31:47.675+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:47.676+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6fe00df492984190265') } 2014-12-23T17:31:48.103+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a7039a8f659f34bfb80d 2014-12-23T17:31:48.103+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:48-5499a7049a8f659f34bfb80e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355908103), what: "moveChunk.start", ns: "benchdb1.COL-61", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:48.159+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-61 based on current shard version 2|0||5499a6fe00df492984190265, current metadata version is 2|0||5499a6fe00df492984190265 2014-12-23T17:31:48.160+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-61 from shard version 2|0||5499a6fe00df492984190265 to shard version 2|1||5499a6fe00df492984190265 2014-12-23T17:31:48.160+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a6fe00df492984190265, took 0ms 2014-12-23T17:31:48.160+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a6fe00df492984190265 2014-12-23T17:31:48.160+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:48.162+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:48.165+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:48.169+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:48.177+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:48.194+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:48.194+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:48.194+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:48.194+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a6fe00df492984190265 2014-12-23T17:31:48.237+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:48.237+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a6fe00df492984190265 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-61' 2014-12-23T17:31:48.453+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:48-5499a7049a8f659f34bfb80f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355908453), what: "moveChunk.commit", ns: "benchdb1.COL-61", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:48.511+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:48.511+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:48.512+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-61 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4341 2014-12-23T17:31:48.512+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-61 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:48.512+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:48.684+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:48.684+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:48-5499a7049a8f659f34bfb810", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355908684), what: "moveChunk.from", ns: "benchdb1.COL-61", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 484, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 317, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:48.740+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-61", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-61-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a6fe00df492984190265') } ntoreturn:1 keyUpdates:0 reslen:37 1064ms 2014-12-23T17:31:48.742+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-61", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-61-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6fe00df492984190265') } 2014-12-23T17:31:49.016+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a7049a8f659f34bfb811 2014-12-23T17:31:49.016+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-61 based on current shard version 3|0||5499a6fe00df492984190265, current metadata version is 3|0||5499a6fe00df492984190265 2014-12-23T17:31:49.017+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-61 from shard version 3|0||5499a6fe00df492984190265 to shard version 3|1||5499a6fe00df492984190265 2014-12-23T17:31:49.017+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a6fe00df492984190265, took 0ms 2014-12-23T17:31:49.017+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a6fe00df492984190265 2014-12-23T17:31:49.199+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:49-5499a7059a8f659f34bfb812", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355909199), what: "split", ns: "benchdb1.COL-61", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a6fe00df492984190265') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a6fe00df492984190265') } } } 2014-12-23T17:31:49.432+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-61/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:49.432+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-61", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-61-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a6fe00df492984190265') } ntoreturn:1 keyUpdates:0 reslen:37 690ms 2014-12-23T17:31:50.832+0000 I INDEX [conn2] build index on: benchdb1.COL-62 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-62" } 2014-12-23T17:31:50.832+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:50.833+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:51.154+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-62 with requested shard version 1|2||5499a70300df49298419026a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:51.154+0000 I SHARDING [conn2] collection benchdb1.COL-62 was previously unsharded, new metadata loaded with shard version 1|2||5499a70300df49298419026a 2014-12-23T17:31:51.154+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a70300df49298419026a, took 0ms 2014-12-23T17:31:51.217+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:51.218+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70300df49298419026a') } 2014-12-23T17:31:51.824+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a7079a8f659f34bfb814 2014-12-23T17:31:51.824+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:51-5499a7079a8f659f34bfb815", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355911824), what: "moveChunk.start", ns: "benchdb1.COL-62", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:51.892+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 based on current shard version 1|2||5499a70300df49298419026a, current metadata version is 1|2||5499a70300df49298419026a 2014-12-23T17:31:51.893+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-62 already up to date (shard version : 1|2||5499a70300df49298419026a, took 0ms) 2014-12-23T17:31:51.893+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a70300df49298419026a 2014-12-23T17:31:51.893+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:51.896+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:51.898+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:51.903+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:51.903+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:51.903+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:51.903+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a70300df49298419026a 2014-12-23T17:31:51.966+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:51.966+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a70300df49298419026a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-62' 2014-12-23T17:31:52.270+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:52-5499a7089a8f659f34bfb816", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355912270), what: "moveChunk.commit", ns: "benchdb1.COL-62", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:52.334+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:52.334+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:52.334+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-62 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4360 2014-12-23T17:31:52.334+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-62 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:52.335+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:52.638+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:52.638+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:52-5499a7089a8f659f34bfb817", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355912638), what: "moveChunk.from", ns: "benchdb1.COL-62", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 674, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 431, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:52.697+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70300df49298419026a') } ntoreturn:1 keyUpdates:0 reslen:37 1479ms 2014-12-23T17:31:52.698+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:52.699+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70300df49298419026a') } 2014-12-23T17:31:53.305+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a7089a8f659f34bfb818 2014-12-23T17:31:53.305+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:53-5499a7099a8f659f34bfb819", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355913305), what: "moveChunk.start", ns: "benchdb1.COL-62", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:31:53.367+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 based on current shard version 2|0||5499a70300df49298419026a, current metadata version is 2|0||5499a70300df49298419026a 2014-12-23T17:31:53.367+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-62 from shard version 2|0||5499a70300df49298419026a to shard version 2|1||5499a70300df49298419026a 2014-12-23T17:31:53.368+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a70300df49298419026a, took 0ms 2014-12-23T17:31:53.368+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a70300df49298419026a 2014-12-23T17:31:53.368+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:53.370+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:53.372+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:53.377+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:53.385+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:53.402+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:53.402+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:53.402+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:53.402+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a70300df49298419026a 2014-12-23T17:31:53.464+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:53.464+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a70300df49298419026a through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-62' 2014-12-23T17:31:53.769+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:53-5499a7099a8f659f34bfb81a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355913769), what: "moveChunk.commit", ns: "benchdb1.COL-62", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:53.834+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:53.834+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:53.834+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-62 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4368 2014-12-23T17:31:53.834+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-62 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:31:53.834+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:54.137+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:54.137+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:54-5499a70a9a8f659f34bfb81b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355914137), what: "moveChunk.from", ns: "benchdb1.COL-62", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 668, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 432, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:31:54.201+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-62", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-62-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70300df49298419026a') } ntoreturn:1 keyUpdates:0 reslen:37 1503ms 2014-12-23T17:31:54.203+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-62", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-62-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a70300df49298419026a') } 2014-12-23T17:31:54.810+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a70a9a8f659f34bfb81c 2014-12-23T17:31:54.810+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 based on current shard version 3|0||5499a70300df49298419026a, current metadata version is 3|0||5499a70300df49298419026a 2014-12-23T17:31:54.811+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-62 from shard version 3|0||5499a70300df49298419026a to shard version 3|1||5499a70300df49298419026a 2014-12-23T17:31:54.811+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a70300df49298419026a, took 0ms 2014-12-23T17:31:54.811+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a70300df49298419026a 2014-12-23T17:31:55.115+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:55-5499a70b9a8f659f34bfb81d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355915115), what: "split", ns: "benchdb1.COL-62", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a70300df49298419026a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a70300df49298419026a') } } } 2014-12-23T17:31:55.506+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-62/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:55.506+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-62", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-62-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a70300df49298419026a') } ntoreturn:1 keyUpdates:0 reslen:37 1303ms 2014-12-23T17:31:58.110+0000 I INDEX [conn2] build index on: benchdb1.COL-63 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-63" } 2014-12-23T17:31:58.110+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:31:58.111+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:31:58.418+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-63 with requested shard version 1|2||5499a70b00df49298419026d, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:31:58.419+0000 I SHARDING [conn2] collection benchdb1.COL-63 was previously unsharded, new metadata loaded with shard version 1|2||5499a70b00df49298419026d 2014-12-23T17:31:58.419+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a70b00df49298419026d, took 0ms 2014-12-23T17:31:58.463+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:58.464+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70b00df49298419026d') } 2014-12-23T17:31:59.071+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a70e9a8f659f34bfb81f 2014-12-23T17:31:59.071+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:59-5499a70f9a8f659f34bfb820", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355919071), what: "moveChunk.start", ns: "benchdb1.COL-63", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:31:59.139+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 based on current shard version 1|2||5499a70b00df49298419026d, current metadata version is 1|2||5499a70b00df49298419026d 2014-12-23T17:31:59.139+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-63 already up to date (shard version : 1|2||5499a70b00df49298419026d, took 0ms) 2014-12-23T17:31:59.139+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a70b00df49298419026d 2014-12-23T17:31:59.140+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:31:59.142+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:59.145+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:59.149+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:31:59.149+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:31:59.150+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:31:59.150+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a70b00df49298419026d 2014-12-23T17:31:59.209+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:31:59.209+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a70b00df49298419026d through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-63' 2014-12-23T17:31:59.513+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:59-5499a70f9a8f659f34bfb821", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355919513), what: "moveChunk.commit", ns: "benchdb1.COL-63", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:31:59.574+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:59.574+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:31:59.574+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-63 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4387 2014-12-23T17:31:59.575+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-63 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:31:59.575+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:31:59.831+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:31:59.831+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:31:59-5499a70f9a8f659f34bfb822", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355919831), what: "moveChunk.from", ns: "benchdb1.COL-63", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 675, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 425, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:31:59.886+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70b00df49298419026d') } ntoreturn:1 keyUpdates:0 reslen:37 1422ms 2014-12-23T17:31:59.888+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:31:59.888+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70b00df49298419026d') } 2014-12-23T17:32:00.495+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a70f9a8f659f34bfb823 2014-12-23T17:32:00.495+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:32:00-5499a7109a8f659f34bfb824", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355920495), what: "moveChunk.start", ns: "benchdb1.COL-63", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:32:00.554+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 based on current shard version 2|0||5499a70b00df49298419026d, current metadata version is 2|0||5499a70b00df49298419026d 2014-12-23T17:32:00.555+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-63 from shard version 2|0||5499a70b00df49298419026d to shard version 2|1||5499a70b00df49298419026d 2014-12-23T17:32:00.555+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a70b00df49298419026d, took 0ms 2014-12-23T17:32:00.555+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a70b00df49298419026d 2014-12-23T17:32:00.555+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:32:00.557+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:32:00.560+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:32:00.564+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:32:00.573+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:32:00.589+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:32:00.589+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:32:00.589+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:32:00.589+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a70b00df49298419026d 2014-12-23T17:32:00.649+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:32:00.649+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a70b00df49298419026d through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-63' 2014-12-23T17:32:00.879+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:32:00-5499a7109a8f659f34bfb825", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355920879), what: "moveChunk.commit", ns: "benchdb1.COL-63", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:32:00.951+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:32:00.951+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:32:00.951+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-63 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4395 2014-12-23T17:32:00.951+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-63 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:32:00.951+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:32:01.204+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:32:01.204+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:32:01-5499a7119a8f659f34bfb826", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355921204), what: "moveChunk.from", ns: "benchdb1.COL-63", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 1, step 4 of 6: 32, step 5 of 6: 361, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:32:01.268+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-63", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-63-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a70b00df49298419026d') } ntoreturn:1 keyUpdates:0 reslen:37 1380ms 2014-12-23T17:32:01.270+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-63", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-63-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a70b00df49298419026d') } 2014-12-23T17:32:01.573+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a7119a8f659f34bfb827 2014-12-23T17:32:01.574+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 based on current shard version 3|0||5499a70b00df49298419026d, current metadata version is 3|0||5499a70b00df49298419026d 2014-12-23T17:32:01.574+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-63 from shard version 3|0||5499a70b00df49298419026d to shard version 3|1||5499a70b00df49298419026d 2014-12-23T17:32:01.574+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a70b00df49298419026d, took 0ms 2014-12-23T17:32:01.574+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a70b00df49298419026d 2014-12-23T17:32:01.764+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:32:01-5499a7119a8f659f34bfb828", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419355921764), what: "split", ns: "benchdb1.COL-63", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a70b00df49298419026d') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a70b00df49298419026d') } } } 2014-12-23T17:32:01.987+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-63/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:32:01.987+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-63", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-63-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a70b00df49298419026d') } ntoreturn:1 keyUpdates:0 reslen:37 716ms 2014-12-23T17:32:11.539+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:32:10 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:32:41.821+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:32:41 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:33:12.427+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:33:11 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:33:43.034+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:33:42 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:34:13.342+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:34:13 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:34:43.949+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:34:43 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:35:12.292+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-0 2014-12-23T17:35:12.359+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a57100df49298419016a, current metadata version is 3|3||5499a57100df49298419016a 2014-12-23T17:35:12.360+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-0, previous version was 0|0||5499a57100df49298419016a, this is a drop 2014-12-23T17:35:12.360+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-0 at shard version 3|3||5499a57100df49298419016a, took 0ms 2014-12-23T17:35:13.421+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-1 2014-12-23T17:35:13.486+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a57800df49298419016d, current metadata version is 3|3||5499a57800df49298419016d 2014-12-23T17:35:13.487+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-1, previous version was 0|0||5499a57800df49298419016d, this is a drop 2014-12-23T17:35:13.487+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-1 at shard version 3|3||5499a57800df49298419016d, took 0ms 2014-12-23T17:35:14.423+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:35:13 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:35:14.577+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-10 2014-12-23T17:35:14.652+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5b600df492984190194, current metadata version is 3|3||5499a5b600df492984190194 2014-12-23T17:35:14.652+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-10, previous version was 0|0||5499a5b600df492984190194, this is a drop 2014-12-23T17:35:14.652+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-10 at shard version 3|3||5499a5b600df492984190194, took 0ms 2014-12-23T17:35:15.773+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-11 2014-12-23T17:35:15.835+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 0|0||000000000000000000000000, current shard version is 3|0||5499a5be00df492984190197, current metadata version is 3|0||5499a5be00df492984190197 2014-12-23T17:35:15.835+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-11, previous version was 0|0||5499a5be00df492984190197, this is a drop 2014-12-23T17:35:15.835+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-11 at shard version 3|0||5499a5be00df492984190197, took 0ms 2014-12-23T17:35:16.752+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-12 2014-12-23T17:35:16.809+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 0|0||000000000000000000000000, current shard version is 4|0||5499a5bf00df49298419019b, current metadata version is 4|0||5499a5bf00df49298419019b 2014-12-23T17:35:16.809+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-12, previous version was 0|0||5499a5bf00df49298419019b, this is a drop 2014-12-23T17:35:16.809+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-12 at shard version 4|0||5499a5bf00df49298419019b, took 0ms 2014-12-23T17:35:17.494+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-13 2014-12-23T17:35:17.553+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5c400df4929841901a1, current metadata version is 3|3||5499a5c400df4929841901a1 2014-12-23T17:35:17.554+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-13, previous version was 0|0||5499a5c400df4929841901a1, this is a drop 2014-12-23T17:35:17.554+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-13 at shard version 3|3||5499a5c400df4929841901a1, took 0ms 2014-12-23T17:35:18.227+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-14 2014-12-23T17:35:18.286+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5cb00df4929841901a4, current metadata version is 3|3||5499a5cb00df4929841901a4 2014-12-23T17:35:18.287+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-14, previous version was 0|0||5499a5cb00df4929841901a4, this is a drop 2014-12-23T17:35:18.287+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-14 at shard version 3|3||5499a5cb00df4929841901a4, took 0ms 2014-12-23T17:35:18.970+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-15 2014-12-23T17:35:19.033+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5d200df4929841901a9, current metadata version is 3|3||5499a5d200df4929841901a9 2014-12-23T17:35:19.034+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-15, previous version was 0|0||5499a5d200df4929841901a9, this is a drop 2014-12-23T17:35:19.034+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-15 at shard version 3|3||5499a5d200df4929841901a9, took 0ms 2014-12-23T17:35:19.972+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-16 2014-12-23T17:35:20.043+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5d900df4929841901ad, current metadata version is 3|3||5499a5d900df4929841901ad 2014-12-23T17:35:20.044+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-16, previous version was 0|0||5499a5d900df4929841901ad, this is a drop 2014-12-23T17:35:20.044+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-16 at shard version 3|3||5499a5d900df4929841901ad, took 0ms 2014-12-23T17:35:21.020+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-17 2014-12-23T17:35:21.097+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5e000df4929841901b1, current metadata version is 3|3||5499a5e000df4929841901b1 2014-12-23T17:35:21.098+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-17, previous version was 0|0||5499a5e000df4929841901b1, this is a drop 2014-12-23T17:35:21.098+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-17 at shard version 3|3||5499a5e000df4929841901b1, took 0ms 2014-12-23T17:35:22.131+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-18 2014-12-23T17:35:22.194+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5e600df4929841901b5, current metadata version is 3|3||5499a5e600df4929841901b5 2014-12-23T17:35:22.195+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-18, previous version was 0|0||5499a5e600df4929841901b5, this is a drop 2014-12-23T17:35:22.195+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-18 at shard version 3|3||5499a5e600df4929841901b5, took 0ms 2014-12-23T17:35:23.277+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-19 2014-12-23T17:35:23.347+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5ec00df4929841901bc, current metadata version is 3|3||5499a5ec00df4929841901bc 2014-12-23T17:35:23.347+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-19, previous version was 0|0||5499a5ec00df4929841901bc, this is a drop 2014-12-23T17:35:23.347+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-19 at shard version 3|3||5499a5ec00df4929841901bc, took 0ms 2014-12-23T17:35:24.426+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-2 2014-12-23T17:35:24.496+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-2 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a57f00df492984190172, current metadata version is 3|3||5499a57f00df492984190172 2014-12-23T17:35:24.497+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-2, previous version was 0|0||5499a57f00df492984190172, this is a drop 2014-12-23T17:35:24.497+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-2 at shard version 3|3||5499a57f00df492984190172, took 0ms 2014-12-23T17:35:25.572+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-20 2014-12-23T17:35:25.635+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5f300df4929841901c0, current metadata version is 3|3||5499a5f300df4929841901c0 2014-12-23T17:35:25.636+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-20, previous version was 0|0||5499a5f300df4929841901c0, this is a drop 2014-12-23T17:35:25.636+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-20 at shard version 3|3||5499a5f300df4929841901c0, took 0ms 2014-12-23T17:35:26.699+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-21 2014-12-23T17:35:26.762+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5f900df4929841901c5, current metadata version is 3|3||5499a5f900df4929841901c5 2014-12-23T17:35:26.762+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-21, previous version was 0|0||5499a5f900df4929841901c5, this is a drop 2014-12-23T17:35:26.762+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-21 at shard version 3|3||5499a5f900df4929841901c5, took 0ms 2014-12-23T17:35:27.835+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-22 2014-12-23T17:35:27.901+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a60100df4929841901c8, current metadata version is 3|3||5499a60100df4929841901c8 2014-12-23T17:35:27.902+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-22, previous version was 0|0||5499a60100df4929841901c8, this is a drop 2014-12-23T17:35:27.902+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-22 at shard version 3|3||5499a60100df4929841901c8, took 0ms 2014-12-23T17:35:29.024+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-23 2014-12-23T17:35:29.091+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-23 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a60700df4929841901cd, current metadata version is 3|3||5499a60700df4929841901cd 2014-12-23T17:35:29.091+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-23, previous version was 0|0||5499a60700df4929841901cd, this is a drop 2014-12-23T17:35:29.091+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-23 at shard version 3|3||5499a60700df4929841901cd, took 0ms 2014-12-23T17:35:30.078+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-24 2014-12-23T17:35:30.149+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-24 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a60e00df4929841901d0, current metadata version is 3|3||5499a60e00df4929841901d0 2014-12-23T17:35:30.149+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-24, previous version was 0|0||5499a60e00df4929841901d0, this is a drop 2014-12-23T17:35:30.149+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-24 at shard version 3|3||5499a60e00df4929841901d0, took 0ms 2014-12-23T17:35:30.782+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-25 2014-12-23T17:35:30.846+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a61500df4929841901d5, current metadata version is 3|3||5499a61500df4929841901d5 2014-12-23T17:35:30.846+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-25, previous version was 0|0||5499a61500df4929841901d5, this is a drop 2014-12-23T17:35:30.846+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-25 at shard version 3|3||5499a61500df4929841901d5, took 0ms 2014-12-23T17:35:31.516+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-26 2014-12-23T17:35:31.573+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-26 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a61c00df4929841901d9, current metadata version is 3|3||5499a61c00df4929841901d9 2014-12-23T17:35:31.573+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-26, previous version was 0|0||5499a61c00df4929841901d9, this is a drop 2014-12-23T17:35:31.574+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-26 at shard version 3|3||5499a61c00df4929841901d9, took 0ms 2014-12-23T17:35:32.287+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-27 2014-12-23T17:35:32.355+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-27 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a62200df4929841901dd, current metadata version is 3|3||5499a62200df4929841901dd 2014-12-23T17:35:32.356+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-27, previous version was 0|0||5499a62200df4929841901dd, this is a drop 2014-12-23T17:35:32.356+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-27 at shard version 3|3||5499a62200df4929841901dd, took 0ms 2014-12-23T17:35:33.489+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-28 2014-12-23T17:35:33.552+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-28 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a62800df4929841901e1, current metadata version is 3|3||5499a62800df4929841901e1 2014-12-23T17:35:33.553+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-28, previous version was 0|0||5499a62800df4929841901e1, this is a drop 2014-12-23T17:35:33.553+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-28 at shard version 3|3||5499a62800df4929841901e1, took 0ms 2014-12-23T17:35:34.631+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-29 2014-12-23T17:35:34.697+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-29 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a62f00df4929841901e5, current metadata version is 3|3||5499a62f00df4929841901e5 2014-12-23T17:35:34.698+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-29, previous version was 0|0||5499a62f00df4929841901e5, this is a drop 2014-12-23T17:35:34.698+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-29 at shard version 3|3||5499a62f00df4929841901e5, took 0ms 2014-12-23T17:35:35.780+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-3 2014-12-23T17:35:35.846+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-3 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a58600df492984190176, current metadata version is 3|3||5499a58600df492984190176 2014-12-23T17:35:35.847+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-3, previous version was 0|0||5499a58600df492984190176, this is a drop 2014-12-23T17:35:35.847+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-3 at shard version 3|3||5499a58600df492984190176, took 0ms 2014-12-23T17:35:36.924+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-30 2014-12-23T17:35:36.988+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-30 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a63500df4929841901e9, current metadata version is 3|3||5499a63500df4929841901e9 2014-12-23T17:35:36.988+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-30, previous version was 0|0||5499a63500df4929841901e9, this is a drop 2014-12-23T17:35:36.988+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-30 at shard version 3|3||5499a63500df4929841901e9, took 0ms 2014-12-23T17:35:38.064+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-31 2014-12-23T17:35:38.129+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-31 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a63c00df4929841901ed, current metadata version is 3|3||5499a63c00df4929841901ed 2014-12-23T17:35:38.129+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-31, previous version was 0|0||5499a63c00df4929841901ed, this is a drop 2014-12-23T17:35:38.129+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-31 at shard version 3|3||5499a63c00df4929841901ed, took 0ms 2014-12-23T17:35:39.190+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-32 2014-12-23T17:35:39.248+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-32 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a64200df4929841901f2, current metadata version is 3|3||5499a64200df4929841901f2 2014-12-23T17:35:39.248+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-32, previous version was 0|0||5499a64200df4929841901f2, this is a drop 2014-12-23T17:35:39.248+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-32 at shard version 3|3||5499a64200df4929841901f2, took 0ms 2014-12-23T17:35:40.341+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-33 2014-12-23T17:35:40.400+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-33 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a64900df4929841901f5, current metadata version is 3|3||5499a64900df4929841901f5 2014-12-23T17:35:40.401+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-33, previous version was 0|0||5499a64900df4929841901f5, this is a drop 2014-12-23T17:35:40.401+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-33 at shard version 3|3||5499a64900df4929841901f5, took 0ms 2014-12-23T17:35:41.440+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-34 2014-12-23T17:35:41.504+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-34 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a64f00df4929841901fa, current metadata version is 3|3||5499a64f00df4929841901fa 2014-12-23T17:35:41.505+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-34, previous version was 0|0||5499a64f00df4929841901fa, this is a drop 2014-12-23T17:35:41.505+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-34 at shard version 3|3||5499a64f00df4929841901fa, took 0ms 2014-12-23T17:35:42.508+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-35 2014-12-23T17:35:42.592+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-35 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a65600df4929841901fd, current metadata version is 3|3||5499a65600df4929841901fd 2014-12-23T17:35:42.593+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-35, previous version was 0|0||5499a65600df4929841901fd, this is a drop 2014-12-23T17:35:42.593+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-35 at shard version 3|3||5499a65600df4929841901fd, took 0ms 2014-12-23T17:35:43.359+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-36 2014-12-23T17:35:43.389+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-36 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a65c00df492984190202, current metadata version is 3|3||5499a65c00df492984190202 2014-12-23T17:35:43.390+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-36, previous version was 0|0||5499a65c00df492984190202, this is a drop 2014-12-23T17:35:43.390+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-36 at shard version 3|3||5499a65c00df492984190202, took 0ms 2014-12-23T17:35:44.019+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-37 2014-12-23T17:35:44.052+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-37 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a66400df492984190205, current metadata version is 3|3||5499a66400df492984190205 2014-12-23T17:35:44.052+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-37, previous version was 0|0||5499a66400df492984190205, this is a drop 2014-12-23T17:35:44.052+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-37 at shard version 3|3||5499a66400df492984190205, took 0ms 2014-12-23T17:35:44.813+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:35:44 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:35:44.967+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-38 2014-12-23T17:35:45.041+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-38 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a66a00df49298419020a, current metadata version is 3|3||5499a66a00df49298419020a 2014-12-23T17:35:45.041+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-38, previous version was 0|0||5499a66a00df49298419020a, this is a drop 2014-12-23T17:35:45.041+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-38 at shard version 3|3||5499a66a00df49298419020a, took 0ms 2014-12-23T17:35:46.112+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-39 2014-12-23T17:35:46.201+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-39 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a67100df49298419020e, current metadata version is 3|3||5499a67100df49298419020e 2014-12-23T17:35:46.202+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-39, previous version was 0|0||5499a67100df49298419020e, this is a drop 2014-12-23T17:35:46.202+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-39 at shard version 3|3||5499a67100df49298419020e, took 0ms 2014-12-23T17:35:47.210+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-4 2014-12-23T17:35:47.281+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-4 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a58d00df49298419017a, current metadata version is 3|3||5499a58d00df49298419017a 2014-12-23T17:35:47.281+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-4, previous version was 0|0||5499a58d00df49298419017a, this is a drop 2014-12-23T17:35:47.281+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-4 at shard version 3|3||5499a58d00df49298419017a, took 0ms 2014-12-23T17:35:48.341+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-40 2014-12-23T17:35:48.402+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-40 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a67700df492984190212, current metadata version is 3|3||5499a67700df492984190212 2014-12-23T17:35:48.403+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-40, previous version was 0|0||5499a67700df492984190212, this is a drop 2014-12-23T17:35:48.403+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-40 at shard version 3|3||5499a67700df492984190212, took 0ms 2014-12-23T17:35:49.460+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-41 2014-12-23T17:35:49.522+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-41 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a67e00df492984190216, current metadata version is 3|3||5499a67e00df492984190216 2014-12-23T17:35:49.523+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-41, previous version was 0|0||5499a67e00df492984190216, this is a drop 2014-12-23T17:35:49.523+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-41 at shard version 3|3||5499a67e00df492984190216, took 0ms 2014-12-23T17:35:50.590+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-42 2014-12-23T17:35:50.651+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-42 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a68400df49298419021a, current metadata version is 3|3||5499a68400df49298419021a 2014-12-23T17:35:50.651+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-42, previous version was 0|0||5499a68400df49298419021a, this is a drop 2014-12-23T17:35:50.651+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-42 at shard version 3|3||5499a68400df49298419021a, took 0ms 2014-12-23T17:35:51.671+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-43 2014-12-23T17:35:51.747+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-43 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a68b00df49298419021e, current metadata version is 3|3||5499a68b00df49298419021e 2014-12-23T17:35:51.748+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-43, previous version was 0|0||5499a68b00df49298419021e, this is a drop 2014-12-23T17:35:51.748+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-43 at shard version 3|3||5499a68b00df49298419021e, took 0ms 2014-12-23T17:35:52.768+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-44 2014-12-23T17:35:52.828+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-44 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a69100df492984190222, current metadata version is 3|3||5499a69100df492984190222 2014-12-23T17:35:52.829+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-44, previous version was 0|0||5499a69100df492984190222, this is a drop 2014-12-23T17:35:52.829+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-44 at shard version 3|3||5499a69100df492984190222, took 0ms 2014-12-23T17:35:53.893+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-45 2014-12-23T17:35:53.956+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-45 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a69700df492984190226, current metadata version is 3|3||5499a69700df492984190226 2014-12-23T17:35:53.957+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-45, previous version was 0|0||5499a69700df492984190226, this is a drop 2014-12-23T17:35:53.957+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-45 at shard version 3|3||5499a69700df492984190226, took 0ms 2014-12-23T17:35:54.923+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-46 2014-12-23T17:35:54.987+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-46 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a69e00df49298419022a, current metadata version is 3|3||5499a69e00df49298419022a 2014-12-23T17:35:54.988+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-46, previous version was 0|0||5499a69e00df49298419022a, this is a drop 2014-12-23T17:35:54.988+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-46 at shard version 3|3||5499a69e00df49298419022a, took 0ms 2014-12-23T17:35:55.678+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-47 2014-12-23T17:35:55.740+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-47 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6a500df49298419022e, current metadata version is 3|3||5499a6a500df49298419022e 2014-12-23T17:35:55.740+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-47, previous version was 0|0||5499a6a500df49298419022e, this is a drop 2014-12-23T17:35:55.741+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-47 at shard version 3|3||5499a6a500df49298419022e, took 0ms 2014-12-23T17:35:56.633+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-48 2014-12-23T17:35:56.700+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-48 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6ab00df492984190232, current metadata version is 3|3||5499a6ab00df492984190232 2014-12-23T17:35:56.700+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-48, previous version was 0|0||5499a6ab00df492984190232, this is a drop 2014-12-23T17:35:56.700+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-48 at shard version 3|3||5499a6ab00df492984190232, took 0ms 2014-12-23T17:35:57.778+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-49 2014-12-23T17:35:57.848+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-49 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6b200df492984190236, current metadata version is 3|3||5499a6b200df492984190236 2014-12-23T17:35:57.848+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-49, previous version was 0|0||5499a6b200df492984190236, this is a drop 2014-12-23T17:35:57.848+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-49 at shard version 3|3||5499a6b200df492984190236, took 0ms 2014-12-23T17:35:58.940+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-5 2014-12-23T17:35:59.002+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-5 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a59400df49298419017f, current metadata version is 3|3||5499a59400df49298419017f 2014-12-23T17:35:59.003+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-5, previous version was 0|0||5499a59400df49298419017f, this is a drop 2014-12-23T17:35:59.003+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-5 at shard version 3|3||5499a59400df49298419017f, took 0ms 2014-12-23T17:36:00.006+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-50 2014-12-23T17:36:00.067+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-50 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6b800df49298419023a, current metadata version is 3|3||5499a6b800df49298419023a 2014-12-23T17:36:00.067+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-50, previous version was 0|0||5499a6b800df49298419023a, this is a drop 2014-12-23T17:36:00.067+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-50 at shard version 3|3||5499a6b800df49298419023a, took 0ms 2014-12-23T17:36:01.131+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-51 2014-12-23T17:36:01.196+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-51 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6bf00df49298419023e, current metadata version is 3|3||5499a6bf00df49298419023e 2014-12-23T17:36:01.197+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-51, previous version was 0|0||5499a6bf00df49298419023e, this is a drop 2014-12-23T17:36:01.197+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-51 at shard version 3|3||5499a6bf00df49298419023e, took 0ms 2014-12-23T17:36:02.279+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-52 2014-12-23T17:36:02.341+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-52 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6c400df492984190242, current metadata version is 3|3||5499a6c400df492984190242 2014-12-23T17:36:02.341+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-52, previous version was 0|0||5499a6c400df492984190242, this is a drop 2014-12-23T17:36:02.341+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-52 at shard version 3|3||5499a6c400df492984190242, took 0ms 2014-12-23T17:36:03.414+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-53 2014-12-23T17:36:03.475+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-53 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6cc00df492984190246, current metadata version is 3|3||5499a6cc00df492984190246 2014-12-23T17:36:03.476+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-53, previous version was 0|0||5499a6cc00df492984190246, this is a drop 2014-12-23T17:36:03.476+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-53 at shard version 3|3||5499a6cc00df492984190246, took 1ms 2014-12-23T17:36:04.563+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-54 2014-12-23T17:36:04.624+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-54 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6d100df49298419024a, current metadata version is 3|3||5499a6d100df49298419024a 2014-12-23T17:36:04.624+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-54, previous version was 0|0||5499a6d100df49298419024a, this is a drop 2014-12-23T17:36:04.624+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-54 at shard version 3|3||5499a6d100df49298419024a, took 0ms 2014-12-23T17:36:05.697+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-55 2014-12-23T17:36:05.759+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-55 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6d900df49298419024e, current metadata version is 3|3||5499a6d900df49298419024e 2014-12-23T17:36:05.760+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-55, previous version was 0|0||5499a6d900df49298419024e, this is a drop 2014-12-23T17:36:05.760+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-55 at shard version 3|3||5499a6d900df49298419024e, took 0ms 2014-12-23T17:36:06.791+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-56 2014-12-23T17:36:06.867+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-56 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6de00df492984190252, current metadata version is 3|3||5499a6de00df492984190252 2014-12-23T17:36:06.868+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-56, previous version was 0|0||5499a6de00df492984190252, this is a drop 2014-12-23T17:36:06.868+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-56 at shard version 3|3||5499a6de00df492984190252, took 0ms 2014-12-23T17:36:07.532+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-57 2014-12-23T17:36:07.596+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-57 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6e500df492984190255, current metadata version is 3|3||5499a6e500df492984190255 2014-12-23T17:36:07.597+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-57, previous version was 0|0||5499a6e500df492984190255, this is a drop 2014-12-23T17:36:07.597+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-57 at shard version 3|3||5499a6e500df492984190255, took 0ms 2014-12-23T17:36:08.697+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-58 2014-12-23T17:36:08.747+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-58 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6eb00df49298419025a, current metadata version is 3|3||5499a6eb00df49298419025a 2014-12-23T17:36:08.748+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-58, previous version was 0|0||5499a6eb00df49298419025a, this is a drop 2014-12-23T17:36:08.748+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-58 at shard version 3|3||5499a6eb00df49298419025a, took 0ms 2014-12-23T17:36:09.825+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-59 2014-12-23T17:36:09.896+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-59 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6f200df49298419025d, current metadata version is 3|3||5499a6f200df49298419025d 2014-12-23T17:36:09.896+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-59, previous version was 0|0||5499a6f200df49298419025d, this is a drop 2014-12-23T17:36:09.896+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-59 at shard version 3|3||5499a6f200df49298419025d, took 0ms 2014-12-23T17:36:10.890+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-6 2014-12-23T17:36:10.952+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a59b00df492984190182, current metadata version is 3|3||5499a59b00df492984190182 2014-12-23T17:36:10.953+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-6, previous version was 0|0||5499a59b00df492984190182, this is a drop 2014-12-23T17:36:10.953+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-6 at shard version 3|3||5499a59b00df492984190182, took 0ms 2014-12-23T17:36:12.030+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-60 2014-12-23T17:36:12.093+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-60 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6f700df492984190262, current metadata version is 3|3||5499a6f700df492984190262 2014-12-23T17:36:12.094+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-60, previous version was 0|0||5499a6f700df492984190262, this is a drop 2014-12-23T17:36:12.094+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-60 at shard version 3|3||5499a6f700df492984190262, took 0ms 2014-12-23T17:36:13.170+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-61 2014-12-23T17:36:13.237+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-61 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a6fe00df492984190265, current metadata version is 3|3||5499a6fe00df492984190265 2014-12-23T17:36:13.237+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-61, previous version was 0|0||5499a6fe00df492984190265, this is a drop 2014-12-23T17:36:13.238+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-61 at shard version 3|3||5499a6fe00df492984190265, took 0ms 2014-12-23T17:36:14.295+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-62 2014-12-23T17:36:14.361+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-62 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a70300df49298419026a, current metadata version is 3|3||5499a70300df49298419026a 2014-12-23T17:36:14.362+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-62, previous version was 0|0||5499a70300df49298419026a, this is a drop 2014-12-23T17:36:14.362+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-62 at shard version 3|3||5499a70300df49298419026a, took 0ms 2014-12-23T17:36:15.293+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:36:14 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:36:15.397+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-63 2014-12-23T17:36:15.481+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-63 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a70b00df49298419026d, current metadata version is 3|3||5499a70b00df49298419026d 2014-12-23T17:36:15.482+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-63, previous version was 0|0||5499a70b00df49298419026d, this is a drop 2014-12-23T17:36:15.482+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-63 at shard version 3|3||5499a70b00df49298419026d, took 0ms 2014-12-23T17:36:16.562+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-7 2014-12-23T17:36:16.631+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5a200df492984190187, current metadata version is 3|3||5499a5a200df492984190187 2014-12-23T17:36:16.632+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-7, previous version was 0|0||5499a5a200df492984190187, this is a drop 2014-12-23T17:36:16.632+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-7 at shard version 3|3||5499a5a200df492984190187, took 0ms 2014-12-23T17:36:17.659+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-8 2014-12-23T17:36:17.743+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5a900df49298419018c, current metadata version is 3|3||5499a5a900df49298419018c 2014-12-23T17:36:17.744+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-8, previous version was 0|0||5499a5a900df49298419018c, this is a drop 2014-12-23T17:36:17.744+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-8 at shard version 3|3||5499a5a900df49298419018c, took 0ms 2014-12-23T17:36:18.520+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-9 2014-12-23T17:36:18.592+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-9 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a5b000df49298419018f, current metadata version is 3|3||5499a5b000df49298419018f 2014-12-23T17:36:18.593+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-9, previous version was 0|0||5499a5b000df49298419018f, this is a drop 2014-12-23T17:36:18.593+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-9 at shard version 3|3||5499a5b000df49298419018f, took 0ms 2014-12-23T17:36:18.993+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:36:18.993+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:36:19.093+0000 I INDEX [conn2] build index on: benchdb1.COL-0 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-0" } 2014-12-23T17:36:19.093+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:36:19.094+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:36:19.373+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 1|2||5499a81000df49298419035c, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:36:19.374+0000 I SHARDING [conn2] collection benchdb1.COL-0 was previously unsharded, new metadata loaded with shard version 1|2||5499a81000df49298419035c 2014-12-23T17:36:19.374+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a81000df49298419035c, took 0ms 2014-12-23T17:36:19.414+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:19.415+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81000df49298419035c') } 2014-12-23T17:36:20.022+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8139a8f659f34bfb82a 2014-12-23T17:36:20.022+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:20-5499a8149a8f659f34bfb82b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356180022), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:36:20.081+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 1|2||5499a81000df49298419035c, current metadata version is 1|2||5499a81000df49298419035c 2014-12-23T17:36:20.082+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-0 already up to date (shard version : 1|2||5499a81000df49298419035c, took 0ms) 2014-12-23T17:36:20.082+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a81000df49298419035c 2014-12-23T17:36:20.082+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:20.085+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:20.088+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:20.092+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:20.092+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:20.093+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:20.093+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a81000df49298419035c 2014-12-23T17:36:20.148+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:20.148+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a81000df49298419035c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:36:20.453+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:20-5499a8149a8f659f34bfb82c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356180453), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:20.506+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:20.506+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:20.506+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4652 2014-12-23T17:36:20.506+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:36:20.506+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:20.809+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:20.809+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:20-5499a8149a8f659f34bfb82d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356180809), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 413, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:36:20.864+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81000df49298419035c') } ntoreturn:1 keyUpdates:0 reslen:37 1449ms 2014-12-23T17:36:20.865+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:20.866+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81000df49298419035c') } 2014-12-23T17:36:21.473+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8149a8f659f34bfb82e 2014-12-23T17:36:21.473+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:21-5499a8159a8f659f34bfb82f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356181473), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:36:21.537+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 2|0||5499a81000df49298419035c, current metadata version is 2|0||5499a81000df49298419035c 2014-12-23T17:36:21.538+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 2|0||5499a81000df49298419035c to shard version 2|1||5499a81000df49298419035c 2014-12-23T17:36:21.538+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a81000df49298419035c, took 0ms 2014-12-23T17:36:21.538+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a81000df49298419035c 2014-12-23T17:36:21.538+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:21.541+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:21.543+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:21.548+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:21.556+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:21.573+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:21.573+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:21.573+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:21.573+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a81000df49298419035c 2014-12-23T17:36:21.626+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:21.626+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a81000df49298419035c through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:36:21.939+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:21-5499a8159a8f659f34bfb830", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356181939), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:22.013+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:22.013+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:22.013+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4660 2014-12-23T17:36:22.013+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:36:22.013+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:22.317+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:22.317+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:22-5499a8169a8f659f34bfb831", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356182317), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 440, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:36:22.370+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81000df49298419035c') } ntoreturn:1 keyUpdates:0 reslen:37 1504ms 2014-12-23T17:36:22.372+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81000df49298419035c') } 2014-12-23T17:36:22.932+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8169a8f659f34bfb832 2014-12-23T17:36:22.932+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 3|0||5499a81000df49298419035c, current metadata version is 3|0||5499a81000df49298419035c 2014-12-23T17:36:22.933+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 3|0||5499a81000df49298419035c to shard version 3|1||5499a81000df49298419035c 2014-12-23T17:36:22.933+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a81000df49298419035c, took 0ms 2014-12-23T17:36:22.933+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a81000df49298419035c 2014-12-23T17:36:23.238+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:23-5499a8179a8f659f34bfb833", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356183238), what: "split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a81000df49298419035c') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a81000df49298419035c') } } } 2014-12-23T17:36:23.587+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:23.587+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81000df49298419035c') } ntoreturn:1 keyUpdates:0 reslen:37 1215ms 2014-12-23T17:36:26.180+0000 I INDEX [conn2] build index on: benchdb1.COL-1 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-1" } 2014-12-23T17:36:26.180+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:36:26.181+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:36:26.502+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 1|2||5499a81700df49298419035f, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:36:26.503+0000 I SHARDING [conn2] collection benchdb1.COL-1 was previously unsharded, new metadata loaded with shard version 1|2||5499a81700df49298419035f 2014-12-23T17:36:26.503+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a81700df49298419035f, took 0ms 2014-12-23T17:36:26.544+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:26.545+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81700df49298419035f') } 2014-12-23T17:36:27.153+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a81a9a8f659f34bfb835 2014-12-23T17:36:27.154+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:27-5499a81b9a8f659f34bfb836", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356187154), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:36:27.218+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||5499a81700df49298419035f, current metadata version is 1|2||5499a81700df49298419035f 2014-12-23T17:36:27.219+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||5499a81700df49298419035f, took 0ms) 2014-12-23T17:36:27.219+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a81700df49298419035f 2014-12-23T17:36:27.220+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:27.222+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:27.225+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:27.229+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:27.229+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:27.230+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:27.230+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a81700df49298419035f 2014-12-23T17:36:27.284+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:27.284+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a81700df49298419035f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:36:27.589+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:27-5499a81b9a8f659f34bfb837", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356187589), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:27.649+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:27.649+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:27.649+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4679 2014-12-23T17:36:27.649+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:36:27.649+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:27.952+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:27.952+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:27-5499a81b9a8f659f34bfb838", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356187952), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 674, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 419, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:36:28.005+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81700df49298419035f') } ntoreturn:1 keyUpdates:0 reslen:37 1460ms 2014-12-23T17:36:28.006+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:28.007+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81700df49298419035f') } 2014-12-23T17:36:28.569+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a81c9a8f659f34bfb839 2014-12-23T17:36:28.569+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:28-5499a81c9a8f659f34bfb83a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356188569), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:36:28.637+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 2|0||5499a81700df49298419035f, current metadata version is 2|0||5499a81700df49298419035f 2014-12-23T17:36:28.638+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 2|0||5499a81700df49298419035f to shard version 2|1||5499a81700df49298419035f 2014-12-23T17:36:28.638+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a81700df49298419035f, took 0ms 2014-12-23T17:36:28.638+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a81700df49298419035f 2014-12-23T17:36:28.638+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:28.641+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:28.643+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:28.648+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:28.656+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:28.672+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:28.672+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:28.673+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:28.673+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a81700df49298419035f 2014-12-23T17:36:28.733+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:28.734+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a81700df49298419035f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:36:29.038+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:29-5499a81d9a8f659f34bfb83b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356189038), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:29.114+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:29.114+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:29.114+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4687 2014-12-23T17:36:29.114+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:36:29.114+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:29.355+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:29.355+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:29-5499a81d9a8f659f34bfb83c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356189355), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 631, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 441, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:36:29.410+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81700df49298419035f') } ntoreturn:1 keyUpdates:0 reslen:37 1403ms 2014-12-23T17:36:29.411+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81700df49298419035f') } 2014-12-23T17:36:30.018+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a81d9a8f659f34bfb83d 2014-12-23T17:36:30.018+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 3|0||5499a81700df49298419035f, current metadata version is 3|0||5499a81700df49298419035f 2014-12-23T17:36:30.019+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 3|0||5499a81700df49298419035f to shard version 3|1||5499a81700df49298419035f 2014-12-23T17:36:30.019+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a81700df49298419035f, took 0ms 2014-12-23T17:36:30.019+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a81700df49298419035f 2014-12-23T17:36:30.323+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:30-5499a81e9a8f659f34bfb83e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356190323), what: "split", ns: "benchdb1.COL-1", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a81700df49298419035f') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a81700df49298419035f') } } } 2014-12-23T17:36:30.735+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:30.735+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81700df49298419035f') } ntoreturn:1 keyUpdates:0 reslen:37 1324ms 2014-12-23T17:36:33.334+0000 I INDEX [conn2] build index on: benchdb1.COL-10 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-10" } 2014-12-23T17:36:33.335+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:36:33.335+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:36:33.661+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 1|2||5499a81e00df492984190364, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:36:33.662+0000 I SHARDING [conn2] collection benchdb1.COL-10 was previously unsharded, new metadata loaded with shard version 1|2||5499a81e00df492984190364 2014-12-23T17:36:33.662+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a81e00df492984190364, took 0ms 2014-12-23T17:36:33.705+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:33.705+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81e00df492984190364') } 2014-12-23T17:36:34.312+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8219a8f659f34bfb840 2014-12-23T17:36:34.312+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:34-5499a8229a8f659f34bfb841", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356194312), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:36:34.378+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 1|2||5499a81e00df492984190364, current metadata version is 1|2||5499a81e00df492984190364 2014-12-23T17:36:34.379+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-10 already up to date (shard version : 1|2||5499a81e00df492984190364, took 0ms) 2014-12-23T17:36:34.379+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a81e00df492984190364 2014-12-23T17:36:34.379+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:34.382+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:34.384+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:34.389+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:34.389+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:34.389+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:34.389+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a81e00df492984190364 2014-12-23T17:36:34.441+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:34.441+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a81e00df492984190364 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:36:34.746+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:34-5499a8229a8f659f34bfb842", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356194746), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:34.801+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:34.801+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:34.801+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4708 2014-12-23T17:36:34.801+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:36:34.801+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:35.104+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:35.104+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:35-5499a8239a8f659f34bfb843", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356195104), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 411, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:36:35.157+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81e00df492984190364') } ntoreturn:1 keyUpdates:0 reslen:37 1452ms 2014-12-23T17:36:35.159+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:35.159+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81e00df492984190364') } 2014-12-23T17:36:35.766+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8239a8f659f34bfb844 2014-12-23T17:36:35.766+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:35-5499a8239a8f659f34bfb845", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356195766), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:36:35.825+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 2|0||5499a81e00df492984190364, current metadata version is 2|0||5499a81e00df492984190364 2014-12-23T17:36:35.826+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-10 from shard version 2|0||5499a81e00df492984190364 to shard version 2|1||5499a81e00df492984190364 2014-12-23T17:36:35.826+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a81e00df492984190364, took 0ms 2014-12-23T17:36:35.826+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a81e00df492984190364 2014-12-23T17:36:35.826+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:35.829+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:35.831+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:35.836+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:35.844+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:35.860+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:35.860+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:35.861+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:35.861+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a81e00df492984190364 2014-12-23T17:36:35.908+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:35.908+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a81e00df492984190364 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:36:36.212+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:36-5499a8249a8f659f34bfb846", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356196212), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:36.277+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:36.277+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:36.277+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4716 2014-12-23T17:36:36.277+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:36:36.277+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:36.580+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:36.580+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:36-5499a8249a8f659f34bfb847", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356196580), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 667, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 416, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:36:36.631+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a81e00df492984190364') } ntoreturn:1 keyUpdates:0 reslen:37 1472ms 2014-12-23T17:36:36.634+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81e00df492984190364') } 2014-12-23T17:36:37.240+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8249a8f659f34bfb848 2014-12-23T17:36:37.240+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 3|0||5499a81e00df492984190364, current metadata version is 3|0||5499a81e00df492984190364 2014-12-23T17:36:37.241+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-10 from shard version 3|0||5499a81e00df492984190364 to shard version 3|1||5499a81e00df492984190364 2014-12-23T17:36:37.241+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a81e00df492984190364, took 0ms 2014-12-23T17:36:37.241+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a81e00df492984190364 2014-12-23T17:36:37.546+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:37-5499a8259a8f659f34bfb849", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356197546), what: "split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a81e00df492984190364') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a81e00df492984190364') } } } 2014-12-23T17:36:37.917+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:37.917+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81e00df492984190364') } ntoreturn:1 keyUpdates:0 reslen:37 1283ms 2014-12-23T17:36:40.294+0000 I INDEX [conn2] build index on: benchdb1.COL-11 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-11" } 2014-12-23T17:36:40.294+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:36:40.295+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:36:40.638+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 1|2||5499a82500df492984190369, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:36:40.639+0000 I SHARDING [conn2] collection benchdb1.COL-11 was previously unsharded, new metadata loaded with shard version 1|2||5499a82500df492984190369 2014-12-23T17:36:40.639+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a82500df492984190369, took 0ms 2014-12-23T17:36:40.678+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:40.679+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a82500df492984190369') } 2014-12-23T17:36:41.285+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8289a8f659f34bfb84b 2014-12-23T17:36:41.286+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:41-5499a8299a8f659f34bfb84c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356201286), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:36:41.349+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 1|2||5499a82500df492984190369, current metadata version is 1|2||5499a82500df492984190369 2014-12-23T17:36:41.350+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-11 already up to date (shard version : 1|2||5499a82500df492984190369, took 0ms) 2014-12-23T17:36:41.350+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a82500df492984190369 2014-12-23T17:36:41.351+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:41.353+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:41.356+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:41.360+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:41.360+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:41.361+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:41.361+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a82500df492984190369 2014-12-23T17:36:41.422+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:41.422+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a82500df492984190369 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T17:36:41.727+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:41-5499a8299a8f659f34bfb84d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356201727), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:41.779+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:41.779+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:41.780+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 4737 2014-12-23T17:36:41.780+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:36:41.780+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:42.083+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:42.083+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:42-5499a82a9a8f659f34bfb84e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356202083), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 419, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:36:42.142+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a82500df492984190369') } ntoreturn:1 keyUpdates:0 reslen:37 1463ms 2014-12-23T17:36:42.144+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:36:42.144+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a82500df492984190369') } 2014-12-23T17:36:42.751+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a82a9a8f659f34bfb84f 2014-12-23T17:36:42.751+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:42-5499a82a9a8f659f34bfb850", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356202751), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:36:42.810+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 2|0||5499a82500df492984190369, current metadata version is 2|0||5499a82500df492984190369 2014-12-23T17:36:42.811+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-11 from shard version 2|0||5499a82500df492984190369 to shard version 2|1||5499a82500df492984190369 2014-12-23T17:36:42.811+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a82500df492984190369, took 0ms 2014-12-23T17:36:42.811+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a82500df492984190369 2014-12-23T17:36:42.811+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:36:42.813+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:42.816+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:42.820+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:42.829+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:42.845+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:36:42.845+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:36:42.845+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:36:42.846+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a82500df492984190369 2014-12-23T17:36:42.901+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:36:42.901+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a82500df492984190369 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T17:36:43.206+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:43-5499a82b9a8f659f34bfb851", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356203206), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:36:43.268+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:43.268+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:36:43.268+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 4745 2014-12-23T17:36:43.268+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:36:43.268+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:36:43.571+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:43.571+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:43-5499a82b9a8f659f34bfb852", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356203571), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 666, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 422, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:36:43.624+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a82500df492984190369') } ntoreturn:1 keyUpdates:0 reslen:37 1480ms 2014-12-23T17:36:43.625+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a82500df492984190369') } 2014-12-23T17:36:44.232+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a82b9a8f659f34bfb853 2014-12-23T17:36:44.232+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 3|0||5499a82500df492984190369, current metadata version is 3|0||5499a82500df492984190369 2014-12-23T17:36:44.233+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-11 from shard version 3|0||5499a82500df492984190369 to shard version 3|1||5499a82500df492984190369 2014-12-23T17:36:44.233+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a82500df492984190369, took 0ms 2014-12-23T17:36:44.233+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a82500df492984190369 2014-12-23T17:36:44.537+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:44-5499a82c9a8f659f34bfb854", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356204537), what: "split", ns: "benchdb1.COL-11", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a82500df492984190369') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a82500df492984190369') } } } 2014-12-23T17:36:44.952+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:44.952+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a82500df492984190369') } ntoreturn:1 keyUpdates:0 reslen:37 1326ms 2014-12-23T17:36:45.816+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:36:45 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:36:49.528+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:36:49.778+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:36:50.247+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:36:50.617+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:36:51.032+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:36:51.418+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:36:51.431+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4624172720304604855 }, { shardkey: -3119957374672154664 } ], shardId: "benchdb1.COL-0-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81000df49298419035c') } 2014-12-23T17:36:51.983+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8339a8f659f34bfb855 2014-12-23T17:36:51.983+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 3|3||5499a81000df49298419035c, current metadata version is 3|3||5499a81000df49298419035c 2014-12-23T17:36:51.984+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 3|3||5499a81000df49298419035c to shard version 3|3||5499a81000df49298419035c 2014-12-23T17:36:51.984+0000 I SHARDING [conn1] collection version was loaded at version 3|7||5499a81000df49298419035c, took 0ms 2014-12-23T17:36:51.984+0000 I SHARDING [conn1] splitChunk accepted at version 3|3||5499a81000df49298419035c 2014-12-23T17:36:52.290+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:52-5499a8349a8f659f34bfb856", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356212290), what: "multi-split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 1, of: 3, chunk: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -4624172720304604855 }, lastmod: Timestamp 3000|8, lastmodEpoch: ObjectId('5499a81000df49298419035c') } } } 2014-12-23T17:36:52.387+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:52-5499a8349a8f659f34bfb857", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356212387), what: "multi-split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 2, of: 3, chunk: { min: { shardkey: -4624172720304604855 }, max: { shardkey: -3119957374672154664 }, lastmod: Timestamp 3000|9, lastmodEpoch: ObjectId('5499a81000df49298419035c') } } } 2014-12-23T17:36:52.432+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:36:52-5499a8349a8f659f34bfb858", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356212432), what: "multi-split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 3, of: 3, chunk: { min: { shardkey: -3119957374672154664 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|10, lastmodEpoch: ObjectId('5499a81000df49298419035c') } } } 2014-12-23T17:36:52.777+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:36:52.777+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4624172720304604855 }, { shardkey: -3119957374672154664 } ], shardId: "benchdb1.COL-0-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81000df49298419035c') } ntoreturn:1 keyUpdates:0 reslen:37 1346ms 2014-12-23T17:36:52.793+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : -6148914691236517204 } -->> { : -4624172720304604855 } 2014-12-23T17:36:52.803+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : -4624172720304604855 } -->> { : -3119957374672154664 } 2014-12-23T17:36:52.811+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : -3119957374672154664 } -->> { : -3074457345618258602 } 2014-12-23T17:36:54.126+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-0 { : -3119957374672154664 } -->> { : -3074457345618258602 } 2014-12-23T17:36:56.603+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-1 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:36:56.924+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-1 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:36:57.515+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-1 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:36:57.918+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-1 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:36:58.358+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-1 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:37:03.905+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:37:04.303+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:37:04.938+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:37:05.444+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:37:05.983+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:37:07.579+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:37:07.593+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4630234165617811310 }, { shardkey: -3122387215967962433 } ], shardId: "benchdb1.COL-10-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81e00df492984190364') } 2014-12-23T17:37:08.136+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8439a8f659f34bfb859 2014-12-23T17:37:08.136+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 3|3||5499a81e00df492984190364, current metadata version is 3|3||5499a81e00df492984190364 2014-12-23T17:37:08.137+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-10 from shard version 3|3||5499a81e00df492984190364 to shard version 3|3||5499a81e00df492984190364 2014-12-23T17:37:08.137+0000 I SHARDING [conn4] collection version was loaded at version 3|10||5499a81e00df492984190364, took 0ms 2014-12-23T17:37:08.137+0000 I SHARDING [conn4] splitChunk accepted at version 3|3||5499a81e00df492984190364 2014-12-23T17:37:08.379+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:37:08-5499a8449a8f659f34bfb85a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356228379), what: "multi-split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 1, of: 3, chunk: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -4630234165617811310 }, lastmod: Timestamp 3000|11, lastmodEpoch: ObjectId('5499a81e00df492984190364') } } } 2014-12-23T17:37:08.432+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:37:08-5499a8449a8f659f34bfb85b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356228432), what: "multi-split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 2, of: 3, chunk: { min: { shardkey: -4630234165617811310 }, max: { shardkey: -3122387215967962433 }, lastmod: Timestamp 3000|12, lastmodEpoch: ObjectId('5499a81e00df492984190364') } } } 2014-12-23T17:37:08.488+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:37:08-5499a8449a8f659f34bfb85c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356228488), what: "multi-split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 3, of: 3, chunk: { min: { shardkey: -3122387215967962433 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|13, lastmodEpoch: ObjectId('5499a81e00df492984190364') } } } 2014-12-23T17:37:08.857+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:37:08.858+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4630234165617811310 }, { shardkey: -3122387215967962433 } ], shardId: "benchdb1.COL-10-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a81e00df492984190364') } ntoreturn:1 keyUpdates:0 reslen:37 1264ms 2014-12-23T17:37:10.307+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : -6148914691236517204 } -->> { : -4630234165617811310 } 2014-12-23T17:37:10.315+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : -4630234165617811310 } -->> { : -3122387215967962433 } 2014-12-23T17:37:10.324+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-10 { : -3122387215967962433 } -->> { : -3074457345618258602 } 2014-12-23T17:37:12.835+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-11 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:37:13.133+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-11 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:37:13.660+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-11 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:37:14.202+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-11 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:37:14.783+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-11 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:37:16.359+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:37:15 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:37:17.886+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 4|1||5499a5bf00df49298419019b, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:37:17.886+0000 W SHARDING [conn2] could not load metadata, collection benchdb1.COL-12 was dropped 2014-12-23T17:37:17.886+0000 I SHARDING [conn2] metadata of collection benchdb1.COL-12 already up to date (shard version : 0|0||000000000000000000000000, took 0ms) 2014-12-23T17:37:18.064+0000 I SHARDING [conn3] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 3|3||5499a81700df49298419035f, current metadata version is 3|3||5499a81700df49298419035f 2014-12-23T17:37:18.065+0000 I SHARDING [conn3] updating metadata for benchdb1.COL-1 from shard version 3|3||5499a81700df49298419035f to shard version 3|3||5499a81700df49298419035f 2014-12-23T17:37:18.065+0000 I SHARDING [conn3] collection version was loaded at version 4|1||5499a81700df49298419035f, took 0ms 2014-12-23T17:37:18.065+0000 I SHARDING [migrateThread] starting receiving-end of migration of chunk { shardkey: -1573960913832059496 } -> { shardkey: -51009502748724932 } for collection benchdb1.COL-1 from ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018 at epoch 5499a81700df49298419035f 2014-12-23T17:37:18.079+0000 I SHARDING [migrateThread] Deleter starting delete for: benchdb1.COL-1 from { shardkey: -1573960913832059496 } -> { shardkey: -51009502748724932 }, with opId: 554080 2014-12-23T17:37:18.079+0000 I SHARDING [migrateThread] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: -1573960913832059496 } -> { shardkey: -51009502748724932 } 2014-12-23T17:37:19.517+0000 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section 2014-12-23T17:37:19.517+0000 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'benchdb1.COL-1' { shardkey: -1573960913832059496 } -> { shardkey: -51009502748724932 } 2014-12-23T17:37:20.120+0000 I NETWORK [initandlisten] connection accepted from 10.238.1.75:48449 #7 (7 connections now open) 2014-12-23T17:37:20.126+0000 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'benchdb1.COL-1' { shardkey: -1573960913832059496 } -> { shardkey: -51009502748724932 } 2014-12-23T17:37:20.128+0000 I SHARDING [migrateThread] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:37:20-5499a8509a8f659f34bfb85d", server: "ip-10-238-44-130", clientAddr: ":27017", time: new Date(1419356240128), what: "moveChunk.to", ns: "benchdb1.COL-1", details: { min: { shardkey: -1573960913832059496 }, max: { shardkey: -51009502748724932 }, step 1 of 5: 13, step 2 of 5: 0, step 3 of 5: 1437, step 4 of 5: 0, step 5 of 5: 611, note: "success" } } 2014-12-23T17:37:20.270+0000 I QUERY [conn7] command admin.$cmd command: _recvChunkCommit { _recvChunkCommit: 1 } ntoreturn:1 keyUpdates:0 reslen:333 149ms 2014-12-23T17:37:22.899+0000 I SHARDING [conn5] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 3|3||5499a82500df492984190369, current metadata version is 3|3||5499a82500df492984190369 2014-12-23T17:37:22.900+0000 I SHARDING [conn5] updating metadata for benchdb1.COL-11 from shard version 3|3||5499a82500df492984190369 to shard version 3|3||5499a82500df492984190369 2014-12-23T17:37:22.900+0000 I SHARDING [conn5] collection version was loaded at version 3|13||5499a82500df492984190369, took 0ms 2014-12-23T17:37:22.900+0000 I SHARDING [migrateThread] starting receiving-end of migration of chunk { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } for collection benchdb1.COL-11 from ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018 at epoch 5499a82500df492984190369 2014-12-23T17:37:22.901+0000 I SHARDING [migrateThread] Deleter starting delete for: benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 }, with opId: 880122 2014-12-23T17:37:22.901+0000 I SHARDING [migrateThread] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } 2014-12-23T17:37:24.887+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 131ms 2014-12-23T17:37:25.447+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 165ms 2014-12-23T17:37:25.843+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 159ms 2014-12-23T17:37:26.079+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 161ms 2014-12-23T17:37:26.204+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 120ms 2014-12-23T17:37:26.344+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 134ms 2014-12-23T17:37:26.654+0000 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section 2014-12-23T17:37:26.654+0000 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'benchdb1.COL-11' { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } 2014-12-23T17:37:27.006+0000 I NETWORK [initandlisten] connection accepted from 10.238.4.248:55203 #8 (8 connections now open) 2014-12-23T17:37:27.010+0000 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'benchdb1.COL-11' { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } 2014-12-23T17:37:27.010+0000 I SHARDING [migrateThread] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:37:27-5499a8579a8f659f34bfb85e", server: "ip-10-238-44-130", clientAddr: ":27017", time: new Date(1419356247010), what: "moveChunk.to", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: 6148914691236517204 }, step 1 of 5: 0, step 2 of 5: 0, step 3 of 5: 3752, step 4 of 5: 0, step 5 of 5: 355, note: "success" } } 2014-12-23T17:37:28.037+0000 I QUERY [conn8] command admin.$cmd command: _recvChunkCommit { _recvChunkCommit: 1 } ntoreturn:1 keyUpdates:0 reslen:334 1028ms 2014-12-23T17:37:31.380+0000 I WRITE [conn2] insert benchdb1.COL-13 query: { _id: ObjectId('5498b35d0315362bd8e5dad0'), shardkey: "vc4HQwrq28CYZkKNqN77cDt2", payload: "GqklH7ng84VtdTmAjMWz0F0jEDmksulOsx4Ex0XV6sarYoWZL1Ctv8P3PaMxXwJQkn1jioNnKsjfYaTjbM0o0PgiSoTTWtvPem4wRAcn88hmtGVXBlpcE2wBPd5PZBWWrHPVJpWuThvwuy3LPZe6Pr..." } ninserted:1 keyUpdates:0 214ms 2014-12-23T17:37:31.386+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 233ms 2014-12-23T17:37:37.089+0000 I WRITE [conn2] insert benchdb1.COL-14 query: { _id: ObjectId('5498b3740315362bd8e8e733'), shardkey: "KYooj4YETtkaOxrz7GZ17NfO", payload: "V55SnRuIsxe65JKRDSLew0A0qIIWaL18aQyGxO9k5Xy5vMeJ34m67gNYmidylaJtW2cof8YSciNfQwu5q1nOAjZnh7EzHq9yuFnFjVpzVFeTwXKbe4Frt5Dpd8CEk3Ufr5cNL8MOY79ALnQw4n40VW..." } ninserted:1 keyUpdates:0 343ms 2014-12-23T17:37:37.095+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-14", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 361ms 2014-12-23T17:37:42.677+0000 I WRITE [conn2] insert benchdb1.COL-15 query: { _id: ObjectId('5498b3890315362bd8ebf343'), shardkey: "3nJrQs44LevDa3kZWIv1K6n5", payload: "PWpiLbrOihkbzwexDyTL5fe3MxSib1rGy4kj7ELjJKA5909yEix4tQAuBC3oHDlOPHQJTuXXBxq3KzbMWHDOLxUldNeqwcGDOBfSdu0Ys9XOUDYe5EVgUybAZVcV0bzr8G1eCn8A0Jvo9CScReqn9K..." } ninserted:1 keyUpdates:0 321ms 2014-12-23T17:37:42.682+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-15", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 338ms 2014-12-23T17:37:43.698+0000 I WRITE [conn2] insert benchdb1.COL-16 query: { _id: ObjectId('5498b38b0315362bd8ec5cd7'), shardkey: "083NDrmxNuvV4IV3CEz5E093", payload: "L8kJpQr0nldHkbKfOEawUpMKgyvYgi9aHCTFrFtCgJajiu4n5zUzAgM2u7CR4exMKg3kMDtmZ56tW4EnuIDWQF52XUNktDIrU4miUPxmWj28cLRlf6B3sooyMLYUfUhGcsWgKFBDo96Il4Smi0qeUf..." } ninserted:1 keyUpdates:0 276ms 2014-12-23T17:37:43.714+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-16", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 292ms 2014-12-23T17:37:46.959+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:37:46 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:37:56.010+0000 I WRITE [conn2] insert benchdb1.COL-18 query: { _id: ObjectId('5498b3b60315362bd8f27539'), shardkey: "4m9FmmOOlBLFl4YkUdlO0YAq", payload: "kgDyIYQ84nWl515BGO95yO6fk5TR2TgcwxCjhzt1IuRWnlzsNtXzgoZAIjlV484Uhn1FdORkbBFff0nY5cKr99hwT6VhrZrMzVXSuB0hVXB0R6BBLS9UWtKlBrRfxuLbCbOrNltHIkP6J31Qkkrb54..." } ninserted:1 keyUpdates:0 339ms 2014-12-23T17:37:56.026+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-18", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 355ms 2014-12-23T17:38:01.499+0000 I WRITE [conn2] insert benchdb1.COL-19 query: { _id: ObjectId('5498b3cb0315362bd8f5816a'), shardkey: "PaSQkqTnN2jekTfFbRfyrlZi", payload: "7Ov9eCVaOvFHJCCEYv9OZjyLEjijUwWxzAKec40mHNXEUCwvw3KSf7YPfgJvbDY6P1dSetVlNEjy9HakxmIGHpoplprap15AbxFdSSWK8F0tzzswINnHY0fCkGjXJT8ETa9ZkuWK0aksscPVrzwjdc..." } ninserted:1 keyUpdates:0 246ms 2014-12-23T17:38:01.515+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-19", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 262ms 2014-12-23T17:38:06.791+0000 I WRITE [conn2] insert benchdb1.COL-2 query: { _id: ObjectId('5498b2640315362bd8c1b229'), shardkey: "UxIxWWOKajRuSuydacHKDqIp", payload: "rtdsg60Uhl0QG6TEEovlnor51WgzTokDC5qCOarJoG7QkkWlRq6Sb712M7YO3fQyvGc8Mm1FNCS9447AzLiEL6vyrR7ra8XW08MWl4dUXgHFaTEQAaTurjLhOrOHftNbwjEAfTsTaFwTbiDwQTL2Yw..." } ninserted:1 keyUpdates:0 106ms 2014-12-23T17:38:06.807+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-2", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 122ms 2014-12-23T17:38:11.689+0000 I WRITE [conn2] insert benchdb1.COL-2 query: { _id: ObjectId('5498b2750315362bd8c45484'), shardkey: "Xbf3GRDL5sq71YvUcknF4dNy", payload: "javks6DcNITvAQATO0xX2XHL2wfDX6e8s1XrXcJFF3FVugUf2XojzGbIdccjHHG10xAHdp2RjWO0aGSgiaiBwr9U075A0IiEzCiwrZzOF8ywjZRbwMDT7ZssQuPHhRbzk32ljwNEHAs37cW1kWsWdw..." } ninserted:1 keyUpdates:0 319ms 2014-12-23T17:38:11.695+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-2", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 335ms 2014-12-23T17:38:17.567+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:38:16 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:38:18.023+0000 I WRITE [conn2] insert benchdb1.COL-21 query: { _id: ObjectId('5498b3f60315362bd8fb99cc'), shardkey: "IYIhnbnePuT63mUt30ZQLi3L", payload: "kb3VzjlBcvIvAlhMaIGssjECSMmHepYzi3GHLaXP5Nb2rha0bxdlprW7H9ZQzAgYX6wlfDL3eHUKP8L5NhctCCP2656sALv9Om8UGNWsaORrbqdI3xegeUKxryr7IbhQI77FQbarYCTEPfmfGTJTfI..." } ninserted:1 keyUpdates:0 302ms 2014-12-23T17:38:18.039+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-21", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 318ms 2014-12-23T17:38:24.493+0000 I WRITE [conn2] insert benchdb1.COL-22 query: { _id: ObjectId('5498b40b0315362bd8fea5fd'), shardkey: "qsQyfSGngaKMnZSRejCh1gnK", payload: "6MFM9phWAwLetDxDwql6irPf99ikAqcXcHIutPpwG16okcw0JEGhTu0P1yEmExxunVdHzUmlF4WhyrHXWsPnIji8aaj0HVOi18peUr9AT6kbWnLxssCZDGzoZxFTiv9NTTnWCeQgeyzDZ3puJI49GZ..." } ninserted:1 keyUpdates:0 428ms 2014-12-23T17:38:24.508+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-22", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 444ms 2014-12-23T17:38:32.579+0000 I WRITE [conn2] insert benchdb1.COL-23 query: { _id: ObjectId('5498b4220315362bd801b22e'), shardkey: "aDHSTv4AfUpuhsRz7bLMfYY7", payload: "Pnoj1HFdjCzffhXVdNMowkOJFSFYWbqLDJSkZom96UaKKi5uzzuZfo7dd9Rrtz9JfLkJghSPJCCc16GP4lROKTF4fijB4R8soCFdtppAgLUvwZjuRAHQ143AJUrHTYq3keAwlNb8AINlRIZxBxgRH1..." } ninserted:1 keyUpdates:0 2033ms 2014-12-23T17:38:32.600+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-23", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 2054ms 2014-12-23T17:38:44.265+0000 I WRITE [conn2] insert benchdb1.COL-25 query: { _id: ObjectId('5498b44c0315362bd807ca90'), shardkey: "9V3umqTV7lSvUs4nvwqXT4i5", payload: "33nB9TuPs20jRd1t5tj7NLTtksb7RLUrwSPvkJwB9WMEq08M5aU5DQT5QO3aoyYcfL9ZvrzLH8OV26zWCUmCUgT752QvXCmIobSNrFV638WoebapfMZ5ffeIFrCP6nRWs9ljZNr9y6QeXo3qDqNyZY..." } ninserted:1 keyUpdates:0 278ms 2014-12-23T17:38:44.281+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-25", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 294ms 2014-12-23T17:38:48.174+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:38:47 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:38:49.948+0000 I WRITE [conn2] insert benchdb1.COL-26 query: { _id: ObjectId('5498b4620315362bd80ad6c1'), shardkey: "0H0wBNx6Ru8DGog5l2YepK9n", payload: "fcWGSQPMML5ufnpqiTiSEjknAz2RHybBTbeTanpQ9C5Kwnzj2mbSeUfJYj9rbwvUF6Yn5iCKKbGJyAChEEZWRzOmBEQGGZ2FPdHsjJXVGgmmdBLxKvoyKY0W5ovspsASC1nZfpo2W84rABeuR7q0WU..." } ninserted:1 keyUpdates:0 552ms 2014-12-23T17:38:49.971+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-26", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 575ms 2014-12-23T17:38:55.614+0000 I WRITE [conn2] insert benchdb1.COL-27 query: { _id: ObjectId('5498b4770315362bd80de2f2'), shardkey: "svVHXmXi9xKG05zU2aCl8ri7", payload: "nMk0LHlevxpiyoPJ0WMylxJfvEjfQuYTWS6Arkj2aSVsV0SkqxkaxKiPhKbi30qRtf5FoJIIsxOq5HcTTfBcXTsFWMikHBVTVhTdtblDERuFX9Nypw07Q97U4vNNDXdqyyYvLZsoLRjZdEcsRQl5Ja..." } ninserted:1 keyUpdates:0 497ms 2014-12-23T17:38:55.629+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-27", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 513ms 2014-12-23T17:39:01.104+0000 I WRITE [conn2] insert benchdb1.COL-28 query: { _id: ObjectId('5498b48c0315362bd810ef23'), shardkey: "Ih8CSFVUKZkLCPArDEeJh2XI", payload: "V22SafBuGzLHyoWdzTSNVOG59W5wQaN2ueJCcYvfi5VXA2yQOcohmzLmEf4ec4MLL0MlXp4O5e2iPx64yw1wpbOZGm6LDwSwngPGjUhOfylJ055PUyqOPqK2SirnanZ5Y0DeolspxCozmZYE4yhJ3j..." } ninserted:1 keyUpdates:0 208ms 2014-12-23T17:39:01.125+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-28", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 230ms 2014-12-23T17:39:06.756+0000 I WRITE [conn2] insert benchdb1.COL-29 query: { _id: ObjectId('5498b4a30315362bd813fb54'), shardkey: "cyJzrsdsdwTXFzpuv0MP9eej", payload: "uOWK6ASjykpTEHyC7C1bQif2nFUohLCOkhO5Ju2f2owNVafQRaysgteC1LhMtPdjptS6URZ2CnwkigRHJxFpSfLOFT3r1bBZEnmZo8eFTy6KQcIgqHv4hUhipwe4Tn5SRnV89CjZrTHVfOdXRHyI5g..." } ninserted:1 keyUpdates:0 493ms 2014-12-23T17:39:06.771+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-29", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 509ms 2014-12-23T17:39:18.436+0000 I WRITE [conn2] insert benchdb1.COL-30 query: { _id: ObjectId('5498b4b70315362bd8170785'), shardkey: "B4q1clX9V5ydXVUqDwxzi0Vd", payload: "LgEAa4pbFFeo6qWRDcaRRzAnWmdahzQjxsMZr9bgAqeUcTWTJTAI8GoS32K2Mbbqora6MNrP7nfn2FARd3F0xjqR0djStlMl95bc4tYFE5suXjPcZkHYLqdgWgG3ZQuoPL4qNTbUAJ31yBoPo7KIhv..." } ninserted:1 keyUpdates:0 265ms 2014-12-23T17:39:18.452+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-30", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 282ms 2014-12-23T17:39:18.780+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:39:18 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:39:23.842+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-31", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 120ms 2014-12-23T17:39:36.166+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:41966 #9 (9 connections now open) 2014-12-23T17:39:36.380+0000 I WRITE [conn2] insert benchdb1.COL-32 query: { _id: ObjectId('5498b4f50315362bd81fc290'), shardkey: "t9SyyZ5ML7ScAvHCyMkgRP98", payload: "3wx9KR1Q4LRfW6n5xpHzHOjuDBnwc3UuiJOwW1ugn3Xwg7DsKX2dDwfT20EV3k1iTNNbVOc522Drm30HGz7PVmRgacwy2kHeOZm8pG5GoObUlxZbXVKLQBp4YWzpLDyZuSdm7SnoX2AU51Q6Oj8Zj0..." } ninserted:1 keyUpdates:0 286ms 2014-12-23T17:39:36.390+0000 I QUERY [conn9] command benchdb1.$cmd command: listCollections { listCollections: 1.0, cursor: {} } ntoreturn:1 keyUpdates:0 reslen:1183 215ms 2014-12-23T17:39:36.391+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-32", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 313ms 2014-12-23T17:39:38.352+0000 I WRITE [conn2] insert benchdb1.COL-33 query: { _id: ObjectId('5498b4f80315362bd8202c18'), shardkey: "poE96Whon3RSxZ33g5BFMZxV", payload: "c3FRraLiQKSxFzn3c59SOuqZvAOS2uXcJIa59KO7ipUAryEdgvGsGQd8RtNnZw4ZVYlA0DK1rRaJWftjveeBbUA7E9R8FykOm3gFJ8myu8knuAErwd48urjYh3Ct8Kxa5wItFG9KNMMRYpJd39zm8E..." } ninserted:1 keyUpdates:0 628ms 2014-12-23T17:39:38.383+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-33", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 659ms 2014-12-23T17:39:39.323+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-0 2014-12-23T17:39:39.555+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 0|0||000000000000000000000000, current shard version is 3|10||5499a81000df49298419035c, current metadata version is 3|10||5499a81000df49298419035c 2014-12-23T17:39:39.555+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-0, previous version was 0|0||5499a81000df49298419035c, this is a drop 2014-12-23T17:39:39.555+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-0 at shard version 3|10||5499a81000df49298419035c, took 0ms 2014-12-23T17:39:40.689+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-1 2014-12-23T17:39:40.781+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a81700df49298419035f, current metadata version is 4|1||5499a81700df49298419035f 2014-12-23T17:39:40.782+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-1, previous version was 0|0||5499a81700df49298419035f, this is a drop 2014-12-23T17:39:40.782+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-1 at shard version 3|3||5499a81700df49298419035f, took 0ms 2014-12-23T17:39:41.930+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-10 2014-12-23T17:39:42.043+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 0|0||000000000000000000000000, current shard version is 3|13||5499a81e00df492984190364, current metadata version is 3|13||5499a81e00df492984190364 2014-12-23T17:39:42.044+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-10, previous version was 0|0||5499a81e00df492984190364, this is a drop 2014-12-23T17:39:42.045+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-10 at shard version 3|13||5499a81e00df492984190364, took 0ms 2014-12-23T17:39:43.176+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-11 2014-12-23T17:39:43.277+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a82500df492984190369, current metadata version is 3|13||5499a82500df492984190369 2014-12-23T17:39:43.278+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-11, previous version was 0|0||5499a82500df492984190369, this is a drop 2014-12-23T17:39:43.278+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-11 at shard version 3|3||5499a82500df492984190369, took 0ms 2014-12-23T17:39:43.673+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:39:44.060+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:39:44.060+0000 I QUERY [conn6] command benchdb1.$cmd command: dropDatabase { dropDatabase: 1 } ntoreturn:1 keyUpdates:0 reslen:59 399ms 2014-12-23T17:39:44.118+0000 I WRITE [conn2] insert benchdb1.COL-33 query: { _id: ObjectId('5498b5080315362bd8229160'), shardkey: "UTFOfT4VHca19PmAsNvF0D52", payload: "U3MnvtD8ET4YSOKqxVoo8um75cxZw5YS4G9L2mmuALxlyr6OWSk0bjZC6FVARTBWcbyjhwPC7pje9tP93yRXgR8QGIYQM5lFBVCsaew0pJE639QlBRAGn875vkElmTDNTXK6k8NM68sXrqYGoeeCVG..." } ninserted:1 keyUpdates:0 440ms 2014-12-23T17:39:44.133+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:39:44.133+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-33", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 456ms 2014-12-23T17:39:44.134+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:39:45.770+0000 I WRITE [conn2] insert benchdb1.COL-34 query: { _id: ObjectId('5498b50d0315362bd8233849'), shardkey: "k4zo93RB3TWSlHiEuTno7QBB", payload: "Yu2OGWxyWW3xkv5W7xEJ09IF5tlN46vG3JrOzMzEtd0g66KK6hHKeUF4DSL9jq6R7TuY6tQPByRttgZCGUBvRMHan9ZNahkepxuwVkF2pZVTvGSuAjr1tsfyzVp0dlHp0YkB8lNEKQvGfXe8hKReIJ..." } ninserted:1 keyUpdates:0 300ms 2014-12-23T17:39:45.785+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-34", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 315ms 2014-12-23T17:39:49.387+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:39:48 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:39:51.212+0000 I WRITE [conn2] insert benchdb1.COL-34 query: { _id: ObjectId('5498b5200315362bd825dac6'), shardkey: "4cpNQ8kDuJn87ihHts1m7AFT", payload: "0m2yoeuDgK2MHOY6DmU8jjDVSNIbYvdjWHmpcr1gYEBgA4ljtC5PQRyLFJMq2P5yLRhxKdJR9wuPymzLrGjxbJOIYo2HUxO0uhcX5BSGWtvBCr1FgWB7wQeV82IGTYutPrulijjsYa3tzXk1zBkYsm..." } ninserted:1 keyUpdates:0 314ms 2014-12-23T17:39:51.218+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-34", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 331ms 2014-12-23T17:39:52.634+0000 I WRITE [conn2] insert benchdb1.COL-35 query: { _id: ObjectId('5498b5230315362bd826447a'), shardkey: "FVYuexbvMaoLvxazgyArGAhz", payload: "WZp7sj4xffLrMuR6S3ExR5LJ1btTO6VVC0eVh4lsPyOanPXT1HQohg62OtnP9AHeA6tVGoeKCjqnfqtunjzxjsmXJ06in41KvmGv3QpNKQ3DPun3LE2jaojMIlQO2hXwMLDy8IizrVYhrJBTArPtYQ..." } ninserted:1 keyUpdates:0 610ms 2014-12-23T17:39:52.651+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-35", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 626ms 2014-12-23T17:39:58.212+0000 I WRITE [conn2] insert benchdb1.COL-36 query: { _id: ObjectId('5498b5380315362bd82950ab'), shardkey: "ihdKSfu3YNzCSZ3Yj5B9a6Ow", payload: "Kt2slC2Belcmh6uUxi0PHMojyShVrbByl0clkNJOrOLZ5BYy88jNIvNHNHIsLoAiH16slEob2GM1W8k8OZXbts0b8szbJsIiWizTdYH7qLBpZhdKu9YVY5HO4dXRItviortzGlIiafHvSN8qY1Y1h6..." } ninserted:1 keyUpdates:0 340ms 2014-12-23T17:39:58.228+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-36", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 356ms 2014-12-23T17:40:03.926+0000 I WRITE [conn9] insert benchdb1.COL-37 query: { _id: ObjectId('5498b54d0315362bd82c5cdc'), shardkey: "y3xNgcGW77PrpJ5ZYkWrodR9", payload: "2iGOF9p3y9G1x56aDQ8KHGRyB4okC17clWFPeLmunQdgjGwQA2lCD5jWWlbckii8R2dYFGx1U7HFfTLJAk07MJG4EYwkN5TKNG6E2m2CMbVK3ArrQVLkqq5Cc5eVGwvwKaxbbUcTeMj0ahVs2lqmFX..." } ninserted:1 keyUpdates:0 455ms 2014-12-23T17:40:03.942+0000 I QUERY [conn9] command benchdb1.$cmd command: insert { insert: "COL-37", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 471ms 2014-12-23T17:40:14.281+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:40:14.290+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:40:14.432+0000 I WRITE [conn9] insert benchdb1.COL-38 query: { _id: ObjectId('5498b5740315362bd831c29d'), shardkey: "p7J090rKeP59xZ0ShBpv6be0", payload: "d1QMtDBNKYHxFor0IdeSENFeqjJoDadceqyU5i6iF1m0nz38QkJrx8yHU6C5oazsfhZCD681BWBLoZUcevGcMCHK0ZokOwdId7PhhM08g237QaOuRknhFGNac07FsfPy5BKRLF8Zz7dMiyXNRNNNdq..." } ninserted:1 keyUpdates:0 146ms 2014-12-23T17:40:14.447+0000 I QUERY [conn9] command benchdb1.$cmd command: insert { insert: "COL-38", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 161ms 2014-12-23T17:40:14.454+0000 I INDEX [conn9] build index on: benchdb1.COL-0 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-0" } 2014-12-23T17:40:14.454+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:40:14.455+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:40:14.782+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 1|2||5499a8fb00df4929841903a4, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:40:14.783+0000 I SHARDING [conn2] collection benchdb1.COL-0 was previously unsharded, new metadata loaded with shard version 1|2||5499a8fb00df4929841903a4 2014-12-23T17:40:14.783+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a8fb00df4929841903a4, took 0ms 2014-12-23T17:40:14.831+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:14.832+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a8fb00df4929841903a4') } 2014-12-23T17:40:15.440+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a8fe9a8f659f34bfb860 2014-12-23T17:40:15.440+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:15-5499a8ff9a8f659f34bfb861", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356415440), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:40:15.736+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 1|2||5499a8fb00df4929841903a4, current metadata version is 1|2||5499a8fb00df4929841903a4 2014-12-23T17:40:15.737+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-0 already up to date (shard version : 1|2||5499a8fb00df4929841903a4, took 0ms) 2014-12-23T17:40:15.737+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a8fb00df4929841903a4 2014-12-23T17:40:15.738+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:15.740+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:15.743+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:15.747+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:15.747+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:15.748+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:15.748+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a8fb00df4929841903a4 2014-12-23T17:40:15.804+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:15.805+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a8fb00df4929841903a4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:40:16.110+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:16-5499a9009a8f659f34bfb862", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356416110), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:16.175+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:16.181+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:16.181+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 12047199 2014-12-23T17:40:16.181+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:40:16.181+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:16.485+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:16.485+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:16-5499a9009a8f659f34bfb863", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356416485), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 905, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 433, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:40:16.539+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a8fb00df4929841903a4') } ntoreturn:1 keyUpdates:0 reslen:37 1707ms 2014-12-23T17:40:16.541+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:16.541+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a8fb00df4929841903a4') } 2014-12-23T17:40:17.150+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9009a8f659f34bfb864 2014-12-23T17:40:17.150+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:17-5499a9019a8f659f34bfb865", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356417150), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:40:17.218+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 2|0||5499a8fb00df4929841903a4, current metadata version is 2|0||5499a8fb00df4929841903a4 2014-12-23T17:40:17.219+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 2|0||5499a8fb00df4929841903a4 to shard version 2|1||5499a8fb00df4929841903a4 2014-12-23T17:40:17.219+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a8fb00df4929841903a4, took 0ms 2014-12-23T17:40:17.219+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a8fb00df4929841903a4 2014-12-23T17:40:17.226+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:17.229+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:17.231+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:17.236+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:17.244+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:17.261+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:17.261+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:17.261+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:17.261+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a8fb00df4929841903a4 2014-12-23T17:40:17.312+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:17.312+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a8fb00df4929841903a4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:40:17.618+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:17-5499a9019a8f659f34bfb866", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356417618), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:17.680+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:17.696+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:17.696+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 12170735 2014-12-23T17:40:17.696+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:40:17.696+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:17.999+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:17.999+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:17-5499a9019a8f659f34bfb867", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356417999), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 678, step 3 of 6: 7, step 4 of 6: 33, step 5 of 6: 434, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:40:18.053+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a8fb00df4929841903a4') } ntoreturn:1 keyUpdates:0 reslen:37 1512ms 2014-12-23T17:40:18.055+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a8fb00df4929841903a4') } 2014-12-23T17:40:18.662+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9029a8f659f34bfb868 2014-12-23T17:40:18.662+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 3|0||5499a8fb00df4929841903a4, current metadata version is 3|0||5499a8fb00df4929841903a4 2014-12-23T17:40:18.663+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 3|0||5499a8fb00df4929841903a4 to shard version 3|1||5499a8fb00df4929841903a4 2014-12-23T17:40:18.663+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a8fb00df4929841903a4, took 0ms 2014-12-23T17:40:18.663+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a8fb00df4929841903a4 2014-12-23T17:40:18.969+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:18-5499a9029a8f659f34bfb869", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356418969), what: "split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a8fb00df4929841903a4') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a8fb00df4929841903a4') } } } 2014-12-23T17:40:19.374+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:19.374+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a8fb00df4929841903a4') } ntoreturn:1 keyUpdates:0 reslen:37 1318ms 2014-12-23T17:40:19.987+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:40:19 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:40:22.072+0000 I INDEX [conn2] build index on: benchdb1.COL-1 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-1" } 2014-12-23T17:40:22.072+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:40:22.073+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:40:22.421+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 1|2||5499a90200df4929841903a9, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:40:22.422+0000 I SHARDING [conn2] collection benchdb1.COL-1 was previously unsharded, new metadata loaded with shard version 1|2||5499a90200df4929841903a9 2014-12-23T17:40:22.422+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a90200df4929841903a9, took 0ms 2014-12-23T17:40:22.474+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:22.474+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90200df4929841903a9') } 2014-12-23T17:40:23.055+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9069a8f659f34bfb86b 2014-12-23T17:40:23.055+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:23-5499a9079a8f659f34bfb86c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356423055), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:40:23.123+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||5499a90200df4929841903a9, current metadata version is 1|2||5499a90200df4929841903a9 2014-12-23T17:40:23.124+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||5499a90200df4929841903a9, took 0ms) 2014-12-23T17:40:23.124+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a90200df4929841903a9 2014-12-23T17:40:23.136+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:23.139+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:23.141+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:23.145+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:23.145+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:23.146+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:23.146+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a90200df4929841903a9 2014-12-23T17:40:23.201+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:23.202+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a90200df4929841903a9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:40:23.507+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:23-5499a9079a8f659f34bfb86d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356423507), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:23.558+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:23.558+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:23.558+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 12594741 2014-12-23T17:40:23.558+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:40:23.558+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:23.862+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:23.863+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:23-5499a9079a8f659f34bfb86e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356423862), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 649, step 3 of 6: 12, step 4 of 6: 8, step 5 of 6: 412, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:40:23.917+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90200df4929841903a9') } ntoreturn:1 keyUpdates:0 reslen:37 1442ms 2014-12-23T17:40:23.918+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:23.919+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90200df4929841903a9') } 2014-12-23T17:40:24.527+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9079a8f659f34bfb86f 2014-12-23T17:40:24.527+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:24-5499a9089a8f659f34bfb870", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356424527), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:40:24.633+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 2|0||5499a90200df4929841903a9, current metadata version is 2|0||5499a90200df4929841903a9 2014-12-23T17:40:24.634+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 2|0||5499a90200df4929841903a9 to shard version 2|1||5499a90200df4929841903a9 2014-12-23T17:40:24.634+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a90200df4929841903a9, took 0ms 2014-12-23T17:40:24.634+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a90200df4929841903a9 2014-12-23T17:40:24.635+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:24.638+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:24.640+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:24.645+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:24.653+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:24.670+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:24.670+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:24.670+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:24.671+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a90200df4929841903a9 2014-12-23T17:40:24.885+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:24.885+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a90200df4929841903a9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-1' 2014-12-23T17:40:25.191+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:25-5499a9099a8f659f34bfb871", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356425191), what: "moveChunk.commit", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:25.272+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:25.288+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:25.288+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 12710520 2014-12-23T17:40:25.288+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-1 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:40:25.288+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:25.592+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:25.592+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:25-5499a9099a8f659f34bfb872", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356425592), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 715, step 3 of 6: 1, step 4 of 6: 34, step 5 of 6: 618, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:40:25.647+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90200df4929841903a9') } ntoreturn:1 keyUpdates:0 reslen:37 1728ms 2014-12-23T17:40:25.649+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a90200df4929841903a9') } 2014-12-23T17:40:26.216+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9099a8f659f34bfb873 2014-12-23T17:40:26.216+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 3|0||5499a90200df4929841903a9, current metadata version is 3|0||5499a90200df4929841903a9 2014-12-23T17:40:26.217+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-1 from shard version 3|0||5499a90200df4929841903a9 to shard version 3|1||5499a90200df4929841903a9 2014-12-23T17:40:26.217+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a90200df4929841903a9, took 0ms 2014-12-23T17:40:26.217+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a90200df4929841903a9 2014-12-23T17:40:26.523+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:26-5499a90a9a8f659f34bfb874", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356426523), what: "split", ns: "benchdb1.COL-1", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a90200df4929841903a9') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a90200df4929841903a9') } } } 2014-12-23T17:40:26.914+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:26.914+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a90200df4929841903a9') } ntoreturn:1 keyUpdates:0 reslen:37 1265ms 2014-12-23T17:40:29.803+0000 I INDEX [conn2] build index on: benchdb1.COL-2 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-2" } 2014-12-23T17:40:29.803+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:40:29.804+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:40:30.118+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-2 with requested shard version 1|2||5499a90a00df4929841903ac, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:40:30.119+0000 I SHARDING [conn2] collection benchdb1.COL-2 was previously unsharded, new metadata loaded with shard version 1|2||5499a90a00df4929841903ac 2014-12-23T17:40:30.119+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a90a00df4929841903ac, took 1ms 2014-12-23T17:40:30.170+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:30.170+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90a00df4929841903ac') } 2014-12-23T17:40:30.739+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a90e9a8f659f34bfb876 2014-12-23T17:40:30.739+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:30-5499a90e9a8f659f34bfb877", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356430739), what: "moveChunk.start", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:40:30.808+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 1|2||5499a90a00df4929841903ac, current metadata version is 1|2||5499a90a00df4929841903ac 2014-12-23T17:40:30.809+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-2 already up to date (shard version : 1|2||5499a90a00df4929841903ac, took 0ms) 2014-12-23T17:40:30.809+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a90a00df4929841903ac 2014-12-23T17:40:30.824+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:30.827+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:30.829+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:30.834+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:30.834+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:30.834+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:30.834+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a90a00df4929841903ac 2014-12-23T17:40:30.891+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:30.891+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a90a00df4929841903ac through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-2' 2014-12-23T17:40:31.155+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:31-5499a90f9a8f659f34bfb878", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356431154), what: "moveChunk.commit", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:31.214+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:31.227+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:31.227+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-2 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 13154689 2014-12-23T17:40:31.227+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-2 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:40:31.227+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:31.470+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:31.470+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:31-5499a90f9a8f659f34bfb879", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356431470), what: "moveChunk.from", ns: "benchdb1.COL-2", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 638, step 3 of 6: 16, step 4 of 6: 8, step 5 of 6: 393, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:40:31.530+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90a00df4929841903ac') } ntoreturn:1 keyUpdates:0 reslen:37 1360ms 2014-12-23T17:40:31.532+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:31.532+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90a00df4929841903ac') } 2014-12-23T17:40:31.785+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a90a00df4929841903ac') } 2014-12-23T17:40:31.787+0000 W SHARDING [conn4] could not acquire collection lock for benchdb1.COL-2 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-2 is taken. 2014-12-23T17:40:31.787+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:31-5499a90f9a8f659f34bfb87b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356431787), what: "moveChunk.from", ns: "benchdb1.COL-2", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-2 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey..." } } 2014-12-23T17:40:31.882+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a90f9a8f659f34bfb87a 2014-12-23T17:40:31.882+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:31-5499a90f9a8f659f34bfb87c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356431882), what: "moveChunk.start", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:40:31.932+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 2|0||5499a90a00df4929841903ac, current metadata version is 2|0||5499a90a00df4929841903ac 2014-12-23T17:40:31.933+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-2 from shard version 2|0||5499a90a00df4929841903ac to shard version 2|1||5499a90a00df4929841903ac 2014-12-23T17:40:31.933+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a90a00df4929841903ac, took 0ms 2014-12-23T17:40:31.933+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a90a00df4929841903ac 2014-12-23T17:40:31.933+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:31.936+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:31.939+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:31.944+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:31.952+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:31.969+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:31.969+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:31.969+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:31.969+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a90a00df4929841903ac 2014-12-23T17:40:32.035+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:32.035+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a90a00df4929841903ac through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-2' 2014-12-23T17:40:32.340+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:32-5499a9109a8f659f34bfb87d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356432340), what: "moveChunk.commit", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:32.407+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:32.407+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:32.407+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-2 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 13259953 2014-12-23T17:40:32.407+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-2 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:40:32.407+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:32.711+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:32.711+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:32-5499a9109a8f659f34bfb87e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356432711), what: "moveChunk.from", ns: "benchdb1.COL-2", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 400, step 3 of 6: 1, step 4 of 6: 34, step 5 of 6: 438, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:40:32.778+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-2", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-2-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a90a00df4929841903ac') } ntoreturn:1 keyUpdates:0 reslen:37 1246ms 2014-12-23T17:40:32.781+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-2", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-2-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a90a00df4929841903ac') } 2014-12-23T17:40:33.344+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9109a8f659f34bfb87f 2014-12-23T17:40:33.344+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-2 based on current shard version 3|0||5499a90a00df4929841903ac, current metadata version is 3|0||5499a90a00df4929841903ac 2014-12-23T17:40:33.344+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-2 from shard version 3|0||5499a90a00df4929841903ac to shard version 3|1||5499a90a00df4929841903ac 2014-12-23T17:40:33.344+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a90a00df4929841903ac, took 0ms 2014-12-23T17:40:33.344+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a90a00df4929841903ac 2014-12-23T17:40:33.609+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:33-5499a9119a8f659f34bfb880", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356433609), what: "split", ns: "benchdb1.COL-2", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a90a00df4929841903ac') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a90a00df4929841903ac') } } } 2014-12-23T17:40:34.238+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-2/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:34.239+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-2", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-2-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a90a00df4929841903ac') } ntoreturn:1 keyUpdates:0 reslen:37 1457ms 2014-12-23T17:40:36.889+0000 I INDEX [conn9] build index on: benchdb1.COL-3 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-3" } 2014-12-23T17:40:36.889+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:40:36.890+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:40:37.209+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-3 with requested shard version 1|2||5499a91100df4929841903b1, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:40:37.209+0000 I SHARDING [conn9] collection benchdb1.COL-3 was previously unsharded, new metadata loaded with shard version 1|2||5499a91100df4929841903b1 2014-12-23T17:40:37.209+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a91100df4929841903b1, took 0ms 2014-12-23T17:40:37.250+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:37.251+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91100df4929841903b1') } 2014-12-23T17:40:37.858+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9159a8f659f34bfb882 2014-12-23T17:40:37.859+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:37-5499a9159a8f659f34bfb883", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356437859), what: "moveChunk.start", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:40:37.929+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 1|2||5499a91100df4929841903b1, current metadata version is 1|2||5499a91100df4929841903b1 2014-12-23T17:40:37.930+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-3 already up to date (shard version : 1|2||5499a91100df4929841903b1, took 0ms) 2014-12-23T17:40:37.930+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a91100df4929841903b1 2014-12-23T17:40:37.930+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:37.933+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:37.935+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:37.940+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:37.940+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:37.940+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:37.940+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a91100df4929841903b1 2014-12-23T17:40:38.048+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:38.048+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a91100df4929841903b1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-3' 2014-12-23T17:40:38.350+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:38-5499a9169a8f659f34bfb884", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356438350), what: "moveChunk.commit", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:38.632+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:38.635+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:38.635+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-3 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 13673095 2014-12-23T17:40:38.635+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-3 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:40:38.635+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:38.939+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:38.939+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:38-5499a9169a8f659f34bfb885", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356438939), what: "moveChunk.from", ns: "benchdb1.COL-3", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 679, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 695, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:40:38.998+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91100df4929841903b1') } ntoreturn:1 keyUpdates:0 reslen:37 1747ms 2014-12-23T17:40:38.999+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:39.000+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91100df4929841903b1') } 2014-12-23T17:40:39.607+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9169a8f659f34bfb886 2014-12-23T17:40:39.608+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:39-5499a9179a8f659f34bfb887", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356439608), what: "moveChunk.start", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:40:39.687+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 2|0||5499a91100df4929841903b1, current metadata version is 2|0||5499a91100df4929841903b1 2014-12-23T17:40:39.688+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-3 from shard version 2|0||5499a91100df4929841903b1 to shard version 2|1||5499a91100df4929841903b1 2014-12-23T17:40:39.688+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a91100df4929841903b1, took 0ms 2014-12-23T17:40:39.689+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a91100df4929841903b1 2014-12-23T17:40:39.701+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:39.704+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:39.706+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:39.711+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:39.719+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:39.736+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:39.736+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:39.736+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:39.736+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a91100df4929841903b1 2014-12-23T17:40:39.829+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:39.830+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a91100df4929841903b1 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-3' 2014-12-23T17:40:40.135+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:40-5499a9189a8f659f34bfb888", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356440135), what: "moveChunk.commit", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:40.418+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:40.444+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:40.444+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-3 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 13800819 2014-12-23T17:40:40.444+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-3 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:40:40.444+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:40.739+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:40.739+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:40-5499a9189a8f659f34bfb889", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356440739), what: "moveChunk.from", ns: "benchdb1.COL-3", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 688, step 3 of 6: 13, step 4 of 6: 33, step 5 of 6: 707, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:40:41.030+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-3", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-3-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91100df4929841903b1') } ntoreturn:1 keyUpdates:0 reslen:37 2030ms 2014-12-23T17:40:41.032+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-3", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-3-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a91100df4929841903b1') } 2014-12-23T17:40:41.600+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9199a8f659f34bfb88a 2014-12-23T17:40:41.600+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-3 based on current shard version 3|0||5499a91100df4929841903b1, current metadata version is 3|0||5499a91100df4929841903b1 2014-12-23T17:40:41.601+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-3 from shard version 3|0||5499a91100df4929841903b1 to shard version 3|1||5499a91100df4929841903b1 2014-12-23T17:40:41.601+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a91100df4929841903b1, took 0ms 2014-12-23T17:40:41.601+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a91100df4929841903b1 2014-12-23T17:40:41.907+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:41-5499a9199a8f659f34bfb88b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356441907), what: "split", ns: "benchdb1.COL-3", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a91100df4929841903b1') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a91100df4929841903b1') } } } 2014-12-23T17:40:42.881+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-3/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:42.881+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-3", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-3-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a91100df4929841903b1') } ntoreturn:1 keyUpdates:0 reslen:37 1849ms 2014-12-23T17:40:45.052+0000 I WRITE [conn2] insert benchdb1.COL-43 query: { _id: ObjectId('5498b5ce0315362bd83ea602'), shardkey: "rFFD3U4Ogh70ytxAhgSHJctb", payload: "EBpP0wOaO2Zz7pvnUsCDVaIVwolX7sMJYE1xTe5ERmAVmPtKuY6WLIzRT8XpTYAKkOkZUDwZVIBbHlSIWjJKQI5ZAUtb3xhYuPD8vB1NBYq6OPZOKesD3OxdZakxog5fW2jHEnucmgKVy3S2uZ33v4..." } ninserted:1 keyUpdates:0 288ms 2014-12-23T17:40:45.075+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-43", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 312ms 2014-12-23T17:40:45.407+0000 I INDEX [conn9] build index on: benchdb1.COL-5 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-5" } 2014-12-23T17:40:45.407+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:40:45.408+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:40:45.722+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-5 with requested shard version 1|2||5499a91a00df4929841903b6, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:40:45.723+0000 I SHARDING [conn2] collection benchdb1.COL-5 was previously unsharded, new metadata loaded with shard version 1|2||5499a91a00df4929841903b6 2014-12-23T17:40:45.723+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a91a00df4929841903b6, took 0ms 2014-12-23T17:40:45.769+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:45.769+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91a00df4929841903b6') } 2014-12-23T17:40:46.376+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a91d9a8f659f34bfb88d 2014-12-23T17:40:46.376+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:46-5499a91e9a8f659f34bfb88e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356446376), what: "moveChunk.start", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:40:46.441+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 1|2||5499a91a00df4929841903b6, current metadata version is 1|2||5499a91a00df4929841903b6 2014-12-23T17:40:46.442+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-5 already up to date (shard version : 1|2||5499a91a00df4929841903b6, took 0ms) 2014-12-23T17:40:46.442+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a91a00df4929841903b6 2014-12-23T17:40:46.453+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:46.455+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:46.458+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:46.462+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:46.462+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:46.463+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:46.463+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a91a00df4929841903b6 2014-12-23T17:40:46.518+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:46.518+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a91a00df4929841903b6 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-5' 2014-12-23T17:40:46.809+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:46-5499a91e9a8f659f34bfb88f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356446809), what: "moveChunk.commit", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:46.863+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:46.863+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:46.863+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-5 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 14163236 2014-12-23T17:40:46.863+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-5 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:40:46.863+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:47.120+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:47.120+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:47-5499a91f9a8f659f34bfb890", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356447120), what: "moveChunk.from", ns: "benchdb1.COL-5", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 11, step 4 of 6: 8, step 5 of 6: 400, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:40:47.182+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91a00df4929841903b6') } ntoreturn:1 keyUpdates:0 reslen:37 1412ms 2014-12-23T17:40:47.183+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:47.184+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91a00df4929841903b6') } 2014-12-23T17:40:47.792+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a91f9a8f659f34bfb891 2014-12-23T17:40:47.792+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:47-5499a91f9a8f659f34bfb892", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356447792), what: "moveChunk.start", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:40:47.860+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 2|0||5499a91a00df4929841903b6, current metadata version is 2|0||5499a91a00df4929841903b6 2014-12-23T17:40:47.861+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-5 from shard version 2|0||5499a91a00df4929841903b6 to shard version 2|1||5499a91a00df4929841903b6 2014-12-23T17:40:47.861+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a91a00df4929841903b6, took 0ms 2014-12-23T17:40:47.861+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a91a00df4929841903b6 2014-12-23T17:40:47.869+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:47.871+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:47.874+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:47.878+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:47.887+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:47.903+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:47.903+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:47.903+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:47.904+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a91a00df4929841903b6 2014-12-23T17:40:47.954+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:47.954+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a91a00df4929841903b6 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-5' 2014-12-23T17:40:48.259+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:48-5499a9209a8f659f34bfb893", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356448259), what: "moveChunk.commit", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:48.318+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:48.326+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:48.326+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-5 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 14270198 2014-12-23T17:40:48.326+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-5 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:40:48.326+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:48.630+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:48.630+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:48-5499a9209a8f659f34bfb894", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356448630), what: "moveChunk.from", ns: "benchdb1.COL-5", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 677, step 3 of 6: 8, step 4 of 6: 33, step 5 of 6: 423, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:40:48.911+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-5", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-5-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a91a00df4929841903b6') } ntoreturn:1 keyUpdates:0 reslen:37 1727ms 2014-12-23T17:40:48.913+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-5", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-5-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a91a00df4929841903b6') } 2014-12-23T17:40:49.516+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9209a8f659f34bfb895 2014-12-23T17:40:49.516+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 3|0||5499a91a00df4929841903b6, current metadata version is 3|0||5499a91a00df4929841903b6 2014-12-23T17:40:49.517+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-5 from shard version 3|0||5499a91a00df4929841903b6 to shard version 3|1||5499a91a00df4929841903b6 2014-12-23T17:40:49.517+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a91a00df4929841903b6, took 0ms 2014-12-23T17:40:49.517+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a91a00df4929841903b6 2014-12-23T17:40:49.822+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:49-5499a9219a8f659f34bfb896", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356449822), what: "split", ns: "benchdb1.COL-5", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a91a00df4929841903b6') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a91a00df4929841903b6') } } } 2014-12-23T17:40:50.161+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-5/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:50.161+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-5", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-5-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a91a00df4929841903b6') } ntoreturn:1 keyUpdates:0 reslen:37 1247ms 2014-12-23T17:40:50.517+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:40:49 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:40:52.671+0000 I INDEX [conn9] build index on: benchdb1.COL-6 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-6" } 2014-12-23T17:40:52.671+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:40:52.671+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:40:52.993+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-6 with requested shard version 1|2||5499a92100df4929841903b9, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:40:52.994+0000 I SHARDING [conn2] collection benchdb1.COL-6 was previously unsharded, new metadata loaded with shard version 1|2||5499a92100df4929841903b9 2014-12-23T17:40:52.994+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a92100df4929841903b9, took 0ms 2014-12-23T17:40:53.044+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:53.044+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92100df4929841903b9') } 2014-12-23T17:40:53.561+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9259a8f659f34bfb898 2014-12-23T17:40:53.561+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:53-5499a9259a8f659f34bfb899", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356453561), what: "moveChunk.start", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:40:53.795+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 1|2||5499a92100df4929841903b9, current metadata version is 1|2||5499a92100df4929841903b9 2014-12-23T17:40:53.796+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-6 already up to date (shard version : 1|2||5499a92100df4929841903b9, took 0ms) 2014-12-23T17:40:53.796+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a92100df4929841903b9 2014-12-23T17:40:53.808+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:53.811+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:53.814+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:53.818+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:53.818+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:53.819+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:53.819+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a92100df4929841903b9 2014-12-23T17:40:53.924+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:53.924+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a92100df4929841903b9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-6' 2014-12-23T17:40:54.143+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:54-5499a9269a8f659f34bfb89a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356454143), what: "moveChunk.commit", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:54.145+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a92100df4929841903b9') } 2014-12-23T17:40:54.145+0000 W SHARDING [conn4] migration already in progress 2014-12-23T17:40:54.145+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:54-5499a9269a8f659f34bfb89b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356454145), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:40:54.192+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:54.199+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:54.199+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-6 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 14669001 2014-12-23T17:40:54.199+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-6 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:40:54.199+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:54.443+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:54.443+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:54-5499a9269a8f659f34bfb89c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356454443), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 752, step 3 of 6: 13, step 4 of 6: 8, step 5 of 6: 380, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:40:54.503+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92100df4929841903b9') } ntoreturn:1 keyUpdates:0 reslen:37 1459ms 2014-12-23T17:40:54.504+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:40:54.504+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92100df4929841903b9') } 2014-12-23T17:40:55.111+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9269a8f659f34bfb89d 2014-12-23T17:40:55.112+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:55-5499a9279a8f659f34bfb89e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356455112), what: "moveChunk.start", ns: "benchdb1.COL-6", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:40:55.175+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 2|0||5499a92100df4929841903b9, current metadata version is 2|0||5499a92100df4929841903b9 2014-12-23T17:40:55.176+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-6 from shard version 2|0||5499a92100df4929841903b9 to shard version 2|1||5499a92100df4929841903b9 2014-12-23T17:40:55.176+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a92100df4929841903b9, took 0ms 2014-12-23T17:40:55.176+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a92100df4929841903b9 2014-12-23T17:40:55.187+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:40:55.189+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:55.192+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:55.196+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:55.205+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:55.221+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:40:55.221+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:40:55.222+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:40:55.222+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a92100df4929841903b9 2014-12-23T17:40:55.281+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:40:55.281+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a92100df4929841903b9 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-6' 2014-12-23T17:40:55.586+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:55-5499a9279a8f659f34bfb89f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356455586), what: "moveChunk.commit", ns: "benchdb1.COL-6", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:40:55.649+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:55.659+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:40:55.659+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-6 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 14758793 2014-12-23T17:40:55.659+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-6 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:40:55.659+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:40:55.962+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:55.963+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:55-5499a9279a8f659f34bfb8a0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356455963), what: "moveChunk.from", ns: "benchdb1.COL-6", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 12, step 4 of 6: 33, step 5 of 6: 437, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:40:56.021+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-6", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-6-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92100df4929841903b9') } ntoreturn:1 keyUpdates:0 reslen:37 1516ms 2014-12-23T17:40:56.023+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92100df4929841903b9') } 2014-12-23T17:40:56.584+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9289a8f659f34bfb8a1 2014-12-23T17:40:56.584+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 3|0||5499a92100df4929841903b9, current metadata version is 3|0||5499a92100df4929841903b9 2014-12-23T17:40:56.585+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-6 from shard version 3|0||5499a92100df4929841903b9 to shard version 3|1||5499a92100df4929841903b9 2014-12-23T17:40:56.585+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a92100df4929841903b9, took 0ms 2014-12-23T17:40:56.585+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a92100df4929841903b9 2014-12-23T17:40:56.889+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:40:56-5499a9289a8f659f34bfb8a2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356456889), what: "split", ns: "benchdb1.COL-6", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a92100df4929841903b9') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a92100df4929841903b9') } } } 2014-12-23T17:40:57.299+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:40:57.299+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-6-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92100df4929841903b9') } ntoreturn:1 keyUpdates:0 reslen:37 1276ms 2014-12-23T17:41:00.106+0000 I INDEX [conn9] build index on: benchdb1.COL-7 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-7" } 2014-12-23T17:41:00.106+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:41:00.107+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:00.429+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-7 with requested shard version 1|2||5499a92900df4929841903be, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:00.429+0000 I SHARDING [conn9] collection benchdb1.COL-7 was previously unsharded, new metadata loaded with shard version 1|2||5499a92900df4929841903be 2014-12-23T17:41:00.430+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a92900df4929841903be, took 0ms 2014-12-23T17:41:00.475+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:00.475+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92900df4929841903be') } 2014-12-23T17:41:01.043+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a92c9a8f659f34bfb8a4 2014-12-23T17:41:01.043+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:01-5499a92d9a8f659f34bfb8a5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356461043), what: "moveChunk.start", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:01.104+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 1|2||5499a92900df4929841903be, current metadata version is 1|2||5499a92900df4929841903be 2014-12-23T17:41:01.105+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-7 already up to date (shard version : 1|2||5499a92900df4929841903be, took 0ms) 2014-12-23T17:41:01.105+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a92900df4929841903be 2014-12-23T17:41:01.105+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:01.108+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:01.111+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:01.115+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:01.115+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:01.115+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:01.115+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a92900df4929841903be 2014-12-23T17:41:01.175+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:01.175+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a92900df4929841903be through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-7' 2014-12-23T17:41:01.480+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:01-5499a92d9a8f659f34bfb8a6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356461480), what: "moveChunk.commit", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:01.532+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:01.547+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:01.547+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-7 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 15132745 2014-12-23T17:41:01.547+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-7 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:01.547+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:01.851+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:01.851+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:01-5499a92d9a8f659f34bfb8a7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356461851), what: "moveChunk.from", ns: "benchdb1.COL-7", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 629, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 432, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:01.909+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92900df4929841903be') } ntoreturn:1 keyUpdates:0 reslen:37 1433ms 2014-12-23T17:41:01.910+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:01.911+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92900df4929841903be') } 2014-12-23T17:41:02.519+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a92d9a8f659f34bfb8a8 2014-12-23T17:41:02.519+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:02-5499a92e9a8f659f34bfb8a9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356462519), what: "moveChunk.start", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:02.577+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 2|0||5499a92900df4929841903be, current metadata version is 2|0||5499a92900df4929841903be 2014-12-23T17:41:02.578+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-7 from shard version 2|0||5499a92900df4929841903be to shard version 2|1||5499a92900df4929841903be 2014-12-23T17:41:02.578+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a92900df4929841903be, took 0ms 2014-12-23T17:41:02.578+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a92900df4929841903be 2014-12-23T17:41:02.578+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:02.581+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:02.583+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:02.588+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:02.596+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:02.613+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:02.613+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:02.615+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:02.615+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a92900df4929841903be 2014-12-23T17:41:02.671+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:02.672+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a92900df4929841903be through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-7' 2014-12-23T17:41:02.977+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:02-5499a92e9a8f659f34bfb8aa", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356462977), what: "moveChunk.commit", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:03.043+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:03.044+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:03.044+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-7 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 15228806 2014-12-23T17:41:03.044+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-7 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:03.044+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:03.348+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:03.348+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:03-5499a92f9a8f659f34bfb8ab", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356463348), what: "moveChunk.from", ns: "benchdb1.COL-7", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 667, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 430, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:03.434+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-7", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-7-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a92900df4929841903be') } ntoreturn:1 keyUpdates:0 reslen:37 1523ms 2014-12-23T17:41:03.436+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-7-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92900df4929841903be') } 2014-12-23T17:41:04.045+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a92f9a8f659f34bfb8ac 2014-12-23T17:41:04.045+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 3|0||5499a92900df4929841903be, current metadata version is 3|0||5499a92900df4929841903be 2014-12-23T17:41:04.045+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-7 from shard version 3|0||5499a92900df4929841903be to shard version 3|1||5499a92900df4929841903be 2014-12-23T17:41:04.046+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a92900df4929841903be, took 0ms 2014-12-23T17:41:04.046+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a92900df4929841903be 2014-12-23T17:41:04.352+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:04-5499a9309a8f659f34bfb8ad", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356464352), what: "split", ns: "benchdb1.COL-7", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a92900df4929841903be') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a92900df4929841903be') } } } 2014-12-23T17:41:04.990+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:04.990+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-7-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92900df4929841903be') } ntoreturn:1 keyUpdates:0 reslen:37 1553ms 2014-12-23T17:41:07.376+0000 I INDEX [conn2] build index on: benchdb1.COL-8 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-8" } 2014-12-23T17:41:07.376+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:41:07.377+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:07.692+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-8 with requested shard version 1|2||5499a93000df4929841903c3, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:07.693+0000 I SHARDING [conn9] collection benchdb1.COL-8 was previously unsharded, new metadata loaded with shard version 1|2||5499a93000df4929841903c3 2014-12-23T17:41:07.693+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a93000df4929841903c3, took 0ms 2014-12-23T17:41:07.736+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:07.737+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93000df4929841903c3') } 2014-12-23T17:41:08.345+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9339a8f659f34bfb8af 2014-12-23T17:41:08.345+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:08-5499a9349a8f659f34bfb8b0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356468345), what: "moveChunk.start", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:08.652+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 1|2||5499a93000df4929841903c3, current metadata version is 1|2||5499a93000df4929841903c3 2014-12-23T17:41:08.653+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-8 already up to date (shard version : 1|2||5499a93000df4929841903c3, took 1ms) 2014-12-23T17:41:08.653+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a93000df4929841903c3 2014-12-23T17:41:08.653+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:08.656+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:08.659+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:08.663+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:08.671+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:08.672+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:08.672+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:08.672+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a93000df4929841903c3 2014-12-23T17:41:08.726+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:08.726+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a93000df4929841903c3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-8' 2014-12-23T17:41:09.032+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:09-5499a9359a8f659f34bfb8b1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356469031), what: "moveChunk.commit", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:09.081+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:09.084+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:09.084+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-8 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 15618260 2014-12-23T17:41:09.085+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-8 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:09.085+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:09.388+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:09.388+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:09-5499a9359a8f659f34bfb8b2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356469388), what: "moveChunk.from", ns: "benchdb1.COL-8", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 916, step 3 of 6: 1, step 4 of 6: 16, step 5 of 6: 412, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:09.451+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93000df4929841903c3') } ntoreturn:1 keyUpdates:0 reslen:37 1714ms 2014-12-23T17:41:09.452+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:09.453+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93000df4929841903c3') } 2014-12-23T17:41:10.060+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9359a8f659f34bfb8b3 2014-12-23T17:41:10.061+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:10-5499a9369a8f659f34bfb8b4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356470061), what: "moveChunk.start", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:10.125+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 2|0||5499a93000df4929841903c3, current metadata version is 2|0||5499a93000df4929841903c3 2014-12-23T17:41:10.126+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-8 from shard version 2|0||5499a93000df4929841903c3 to shard version 2|1||5499a93000df4929841903c3 2014-12-23T17:41:10.126+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a93000df4929841903c3, took 0ms 2014-12-23T17:41:10.126+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a93000df4929841903c3 2014-12-23T17:41:10.138+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:10.140+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:10.143+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:10.147+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:10.156+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:10.172+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:10.172+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:10.173+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:10.173+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a93000df4929841903c3 2014-12-23T17:41:10.229+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:10.229+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a93000df4929841903c3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-8' 2014-12-23T17:41:10.535+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:10-5499a9369a8f659f34bfb8b5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356470535), what: "moveChunk.commit", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:10.598+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:10.603+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:10.603+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-8 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 15737789 2014-12-23T17:41:10.603+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-8 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:10.603+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:10.892+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:10.892+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:10-5499a9369a8f659f34bfb8b6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356470892), what: "moveChunk.from", ns: "benchdb1.COL-8", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 12, step 4 of 6: 33, step 5 of 6: 430, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:10.956+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-8", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-8-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93000df4929841903c3') } ntoreturn:1 keyUpdates:0 reslen:37 1504ms 2014-12-23T17:41:10.958+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-8-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93000df4929841903c3') } 2014-12-23T17:41:11.566+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9369a8f659f34bfb8b7 2014-12-23T17:41:11.566+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 3|0||5499a93000df4929841903c3, current metadata version is 3|0||5499a93000df4929841903c3 2014-12-23T17:41:11.567+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-8 from shard version 3|0||5499a93000df4929841903c3 to shard version 3|1||5499a93000df4929841903c3 2014-12-23T17:41:11.567+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a93000df4929841903c3, took 0ms 2014-12-23T17:41:11.567+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a93000df4929841903c3 2014-12-23T17:41:11.872+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:11-5499a9379a8f659f34bfb8b8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356471872), what: "split", ns: "benchdb1.COL-8", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a93000df4929841903c3') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a93000df4929841903c3') } } } 2014-12-23T17:41:12.260+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:12.261+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-8-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93000df4929841903c3') } ntoreturn:1 keyUpdates:0 reslen:37 1302ms 2014-12-23T17:41:15.159+0000 I INDEX [conn2] build index on: benchdb1.COL-9 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-9" } 2014-12-23T17:41:15.159+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:41:15.160+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:15.485+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-9 with requested shard version 1|2||5499a93800df4929841903c6, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:15.485+0000 I SHARDING [conn9] collection benchdb1.COL-9 was previously unsharded, new metadata loaded with shard version 1|2||5499a93800df4929841903c6 2014-12-23T17:41:15.486+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a93800df4929841903c6, took 0ms 2014-12-23T17:41:15.527+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:15.528+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93800df4929841903c6') } 2014-12-23T17:41:16.007+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a93b9a8f659f34bfb8ba 2014-12-23T17:41:16.007+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:16-5499a93c9a8f659f34bfb8bb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356476007), what: "moveChunk.start", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:16.071+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 1|2||5499a93800df4929841903c6, current metadata version is 1|2||5499a93800df4929841903c6 2014-12-23T17:41:16.072+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-9 already up to date (shard version : 1|2||5499a93800df4929841903c6, took 0ms) 2014-12-23T17:41:16.072+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a93800df4929841903c6 2014-12-23T17:41:16.072+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:16.075+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:16.078+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:16.082+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:16.082+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:16.083+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:16.083+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a93800df4929841903c6 2014-12-23T17:41:16.146+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:16.146+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a93800df4929841903c6 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-9' 2014-12-23T17:41:16.324+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:16-5499a93c9a8f659f34bfb8bc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356476324), what: "moveChunk.commit", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:16.355+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:16.355+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:16.355+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-9 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 16115806 2014-12-23T17:41:16.355+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-9 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:16.355+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:16.518+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:16.518+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:16-5499a93c9a8f659f34bfb8bd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356476518), what: "moveChunk.from", ns: "benchdb1.COL-9", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 543, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 272, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:16.557+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a93800df4929841903c6') } 2014-12-23T17:41:16.575+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93800df4929841903c6') } ntoreturn:1 keyUpdates:0 reslen:37 1047ms 2014-12-23T17:41:16.576+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:16.576+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93800df4929841903c6') } 2014-12-23T17:41:16.879+0000 I SHARDING [conn1] could not acquire lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' (another update won) 2014-12-23T17:41:16.879+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' was not acquired. 2014-12-23T17:41:16.879+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-9 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-9 is taken. 2014-12-23T17:41:16.879+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:16-5499a93c9a8f659f34bfb8c0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356476879), what: "moveChunk.from", ns: "benchdb1.COL-9", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-9 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey..." } } 2014-12-23T17:41:16.947+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93800df4929841903c6') } ntoreturn:1 keyUpdates:0 reslen:245 371ms 2014-12-23T17:41:16.949+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-9-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93800df4929841903c6') } 2014-12-23T17:41:16.950+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-9 to split chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for splitting chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-9 is taken. 2014-12-23T17:41:16.953+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93800df4929841903c6') } 2014-12-23T17:41:16.954+0000 W SHARDING [conn1] could not acquire collection lock for benchdb1.COL-9 to split chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for splitting chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-9 is taken. 2014-12-23T17:41:16.965+0000 I INDEX [conn2] build index on: benchdb1.COL-10 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-10" } 2014-12-23T17:41:16.965+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:41:16.966+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:17.021+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a93c9a8f659f34bfb8be 2014-12-23T17:41:17.021+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:17-5499a93d9a8f659f34bfb8c2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356477021), what: "moveChunk.start", ns: "benchdb1.COL-9", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:17.117+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 2|0||5499a93800df4929841903c6, current metadata version is 2|0||5499a93800df4929841903c6 2014-12-23T17:41:17.118+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-9 from shard version 2|0||5499a93800df4929841903c6 to shard version 2|1||5499a93800df4929841903c6 2014-12-23T17:41:17.118+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a93800df4929841903c6, took 0ms 2014-12-23T17:41:17.118+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a93800df4929841903c6 2014-12-23T17:41:17.118+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:41:17.121+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:17.123+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:17.128+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:17.136+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:17.153+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:17.153+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:41:17.153+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:41:17.153+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a93800df4929841903c6 2014-12-23T17:41:17.245+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:17.245+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a93800df4929841903c6 through { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } for collection 'benchdb1.COL-9' 2014-12-23T17:41:17.415+0000 I SHARDING [conn2] waiting till out of critical section 2014-12-23T17:41:17.465+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:17-5499a93d9a8f659f34bfb8c3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356477465), what: "moveChunk.commit", ns: "benchdb1.COL-9", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:17.466+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 1|2||5499a93900df4929841903ca, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:17.467+0000 I SHARDING [conn2] collection benchdb1.COL-10 was previously unsharded, new metadata loaded with shard version 1|2||5499a93900df4929841903ca 2014-12-23T17:41:17.467+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a93900df4929841903ca, took 0ms 2014-12-23T17:41:17.511+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:17.511+0000 I SHARDING [conn4] forking for cleanup of chunk data 2014-12-23T17:41:17.511+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:17.511+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-9 from { shardkey: MinKey } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T17:41:17.511+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-9 from { shardkey: MinKey } -> { shardkey: -3074457345618258602 } 2014-12-23T17:41:17.550+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:17.550+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93900df4929841903ca') } 2014-12-23T17:41:17.752+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:17.752+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:17-5499a93d9a8f659f34bfb8c5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356477752), what: "moveChunk.from", ns: "benchdb1.COL-9", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 561, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 357, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:17.819+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-9", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-9-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a93800df4929841903c6') } ntoreturn:1 keyUpdates:0 reslen:37 1262ms 2014-12-23T17:41:18.069+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a93d9a8f659f34bfb8c4 2014-12-23T17:41:18.069+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:18-5499a93e9a8f659f34bfb8c6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356478069), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:18.128+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 1|2||5499a93900df4929841903ca, current metadata version is 1|2||5499a93900df4929841903ca 2014-12-23T17:41:18.129+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-10 already up to date (shard version : 1|2||5499a93900df4929841903ca, took 0ms) 2014-12-23T17:41:18.129+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a93900df4929841903ca 2014-12-23T17:41:18.141+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:18.144+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:18.146+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:18.151+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:18.151+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:18.151+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:18.151+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a93900df4929841903ca 2014-12-23T17:41:18.212+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:18.212+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a93900df4929841903ca through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:41:18.517+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:18-5499a93e9a8f659f34bfb8c7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356478516), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:18.588+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:18.589+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:18.589+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 16219806 2014-12-23T17:41:18.589+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:18.589+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:18.892+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:18.892+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:18-5499a93e9a8f659f34bfb8c8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356478892), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 578, step 3 of 6: 13, step 4 of 6: 8, step 5 of 6: 437, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:18.946+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93900df4929841903ca') } ntoreturn:1 keyUpdates:0 reslen:37 1396ms 2014-12-23T17:41:18.948+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:18.948+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93900df4929841903ca') } 2014-12-23T17:41:19.480+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a93e9a8f659f34bfb8c9 2014-12-23T17:41:19.480+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:19-5499a93f9a8f659f34bfb8ca", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356479480), what: "moveChunk.start", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:19.638+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 2|0||5499a93900df4929841903ca, current metadata version is 2|0||5499a93900df4929841903ca 2014-12-23T17:41:19.639+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-10 from shard version 2|0||5499a93900df4929841903ca to shard version 2|1||5499a93900df4929841903ca 2014-12-23T17:41:19.639+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a93900df4929841903ca, took 0ms 2014-12-23T17:41:19.639+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a93900df4929841903ca 2014-12-23T17:41:19.657+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:19.662+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:19.665+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:19.681+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:19.690+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:19.707+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:19.707+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:19.707+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:19.707+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a93900df4929841903ca 2014-12-23T17:41:19.850+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:19.850+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a93900df4929841903ca through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-10' 2014-12-23T17:41:20.048+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:20-5499a9409a8f659f34bfb8cb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356480048), what: "moveChunk.commit", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:20.096+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:20.099+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:20.099+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 16307863 2014-12-23T17:41:20.099+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-10 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:20.099+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:20.313+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:20.313+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:20-5499a9409a8f659f34bfb8cc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356480313), what: "moveChunk.from", ns: "benchdb1.COL-10", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 690, step 3 of 6: 21, step 4 of 6: 46, step 5 of 6: 391, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:20.363+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-10", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-10-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a93900df4929841903ca') } ntoreturn:1 keyUpdates:0 reslen:37 1415ms 2014-12-23T17:41:20.365+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93900df4929841903ca') } 2014-12-23T17:41:20.892+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9409a8f659f34bfb8cd 2014-12-23T17:41:20.892+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-10 based on current shard version 3|0||5499a93900df4929841903ca, current metadata version is 3|0||5499a93900df4929841903ca 2014-12-23T17:41:20.893+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-10 from shard version 3|0||5499a93900df4929841903ca to shard version 3|1||5499a93900df4929841903ca 2014-12-23T17:41:20.893+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a93900df4929841903ca, took 0ms 2014-12-23T17:41:20.893+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a93900df4929841903ca 2014-12-23T17:41:20.994+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:41:20 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:41:21.198+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:21-5499a9419a8f659f34bfb8ce", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356481198), what: "split", ns: "benchdb1.COL-10", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a93900df4929841903ca') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a93900df4929841903ca') } } } 2014-12-23T17:41:21.556+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-10/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:21.556+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-10", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-10-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93900df4929841903ca') } ntoreturn:1 keyUpdates:0 reslen:37 1191ms 2014-12-23T17:41:24.368+0000 I INDEX [conn9] build index on: benchdb1.COL-11 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-11" } 2014-12-23T17:41:24.368+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:41:24.368+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:24.391+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-49", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 102ms 2014-12-23T17:41:24.674+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 1|2||5499a94100df4929841903d0, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:24.674+0000 I SHARDING [conn2] collection benchdb1.COL-11 was previously unsharded, new metadata loaded with shard version 1|2||5499a94100df4929841903d0 2014-12-23T17:41:24.674+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a94100df4929841903d0, took 0ms 2014-12-23T17:41:24.720+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:24.720+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94100df4929841903d0') } 2014-12-23T17:41:25.327+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9449a8f659f34bfb8d0 2014-12-23T17:41:25.328+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:25-5499a9459a8f659f34bfb8d1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356485328), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:25.395+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 1|2||5499a94100df4929841903d0, current metadata version is 1|2||5499a94100df4929841903d0 2014-12-23T17:41:25.396+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-11 already up to date (shard version : 1|2||5499a94100df4929841903d0, took 0ms) 2014-12-23T17:41:25.396+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a94100df4929841903d0 2014-12-23T17:41:25.408+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:25.410+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:25.413+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:25.417+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:25.417+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:25.418+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:25.418+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a94100df4929841903d0 2014-12-23T17:41:25.473+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:25.473+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a94100df4929841903d0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T17:41:25.778+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:25-5499a9459a8f659f34bfb8d2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356485778), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:25.841+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:25.848+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:25.848+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 16667540 2014-12-23T17:41:25.848+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:25.848+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:26.152+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:26.152+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:26-5499a9469a8f659f34bfb8d3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356486152), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 675, step 3 of 6: 12, step 4 of 6: 8, step 5 of 6: 430, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:26.202+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94100df4929841903d0') } ntoreturn:1 keyUpdates:0 reslen:37 1482ms 2014-12-23T17:41:26.204+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:26.204+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94100df4929841903d0') } 2014-12-23T17:41:26.812+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9469a8f659f34bfb8d4 2014-12-23T17:41:26.812+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:26-5499a9469a8f659f34bfb8d5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356486812), what: "moveChunk.start", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:26.879+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 2|0||5499a94100df4929841903d0, current metadata version is 2|0||5499a94100df4929841903d0 2014-12-23T17:41:26.880+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-11 from shard version 2|0||5499a94100df4929841903d0 to shard version 2|1||5499a94100df4929841903d0 2014-12-23T17:41:26.880+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a94100df4929841903d0, took 0ms 2014-12-23T17:41:26.880+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a94100df4929841903d0 2014-12-23T17:41:26.880+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:26.882+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:26.885+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:26.889+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:26.898+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:26.914+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:26.914+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:26.915+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:26.915+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a94100df4929841903d0 2014-12-23T17:41:26.969+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:26.969+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a94100df4929841903d0 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-11' 2014-12-23T17:41:27.282+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:27-5499a9479a8f659f34bfb8d6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356487282), what: "moveChunk.commit", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:27.339+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:27.341+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:27.341+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 16776686 2014-12-23T17:41:27.342+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-11 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:27.342+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:27.645+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:27.645+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:27-5499a9479a8f659f34bfb8d7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356487645), what: "moveChunk.from", ns: "benchdb1.COL-11", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 675, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 427, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:27.746+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-11", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-11-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94100df4929841903d0') } ntoreturn:1 keyUpdates:0 reslen:37 1542ms 2014-12-23T17:41:27.748+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a94100df4929841903d0') } 2014-12-23T17:41:28.309+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9479a8f659f34bfb8d8 2014-12-23T17:41:28.309+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-11 based on current shard version 3|0||5499a94100df4929841903d0, current metadata version is 3|0||5499a94100df4929841903d0 2014-12-23T17:41:28.310+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-11 from shard version 3|0||5499a94100df4929841903d0 to shard version 3|1||5499a94100df4929841903d0 2014-12-23T17:41:28.310+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a94100df4929841903d0, took 0ms 2014-12-23T17:41:28.310+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a94100df4929841903d0 2014-12-23T17:41:28.618+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:28-5499a9489a8f659f34bfb8d9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356488618), what: "split", ns: "benchdb1.COL-11", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a94100df4929841903d0') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a94100df4929841903d0') } } } 2014-12-23T17:41:29.038+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-11/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:29.038+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-11", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-11-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a94100df4929841903d0') } ntoreturn:1 keyUpdates:0 reslen:37 1290ms 2014-12-23T17:41:30.388+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-5 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:41:30.639+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-5 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:41:31.131+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-5 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:41:31.541+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-5 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:41:31.617+0000 I INDEX [conn9] build index on: benchdb1.COL-12 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-12" } 2014-12-23T17:41:31.617+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:41:31.619+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:31.935+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 1|2||5499a94800df4929841903d4, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:31.935+0000 I SHARDING [conn9] collection benchdb1.COL-12 was previously unsharded, new metadata loaded with shard version 1|2||5499a94800df4929841903d4 2014-12-23T17:41:31.936+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a94800df4929841903d4, took 0ms 2014-12-23T17:41:31.981+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:31.982+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94800df4929841903d4') } 2014-12-23T17:41:32.218+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-5 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:41:32.533+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a94b9a8f659f34bfb8db 2014-12-23T17:41:32.533+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:32-5499a94c9a8f659f34bfb8dc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356492533), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:32.633+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 1|2||5499a94800df4929841903d4, current metadata version is 1|2||5499a94800df4929841903d4 2014-12-23T17:41:32.633+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-12 already up to date (shard version : 1|2||5499a94800df4929841903d4, took 0ms) 2014-12-23T17:41:32.633+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a94800df4929841903d4 2014-12-23T17:41:32.634+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:32.636+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:32.639+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:32.643+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:32.652+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:32.652+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:32.652+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:32.652+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a94800df4929841903d4 2014-12-23T17:41:32.719+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:32.719+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a94800df4929841903d4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T17:41:33.028+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:33-5499a94d9a8f659f34bfb8dd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356493028), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:33.108+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:33.108+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:33.108+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-12 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 17035916 2014-12-23T17:41:33.108+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:33.108+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:33.313+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:33.313+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:33-5499a94d9a8f659f34bfb8de", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356493313), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 651, step 3 of 6: 1, step 4 of 6: 16, step 5 of 6: 455, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:33.442+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94800df4929841903d4') } ntoreturn:1 keyUpdates:0 reslen:37 1461ms 2014-12-23T17:41:33.446+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:33.446+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94800df4929841903d4') } 2014-12-23T17:41:33.979+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a94d9a8f659f34bfb8df 2014-12-23T17:41:33.980+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:33-5499a94d9a8f659f34bfb8e0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356493980), what: "moveChunk.start", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:34.042+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 2|0||5499a94800df4929841903d4, current metadata version is 2|0||5499a94800df4929841903d4 2014-12-23T17:41:34.043+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-12 from shard version 2|0||5499a94800df4929841903d4 to shard version 2|1||5499a94800df4929841903d4 2014-12-23T17:41:34.043+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a94800df4929841903d4, took 0ms 2014-12-23T17:41:34.043+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a94800df4929841903d4 2014-12-23T17:41:34.043+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:34.046+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:34.048+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:34.053+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:34.061+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:34.077+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:34.077+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:34.078+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:34.078+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a94800df4929841903d4 2014-12-23T17:41:34.129+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:34.129+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a94800df4929841903d4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-12' 2014-12-23T17:41:34.434+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:34-5499a94e9a8f659f34bfb8e1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356494434), what: "moveChunk.commit", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:34.504+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:34.504+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:34.504+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 17057388 2014-12-23T17:41:34.504+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-12 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:34.504+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:34.735+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:34.735+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:34-5499a94e9a8f659f34bfb8e2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356494735), what: "moveChunk.from", ns: "benchdb1.COL-12", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 597, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 426, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:34.787+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-12", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-12-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94800df4929841903d4') } ntoreturn:1 keyUpdates:0 reslen:37 1341ms 2014-12-23T17:41:34.791+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a94800df4929841903d4') } 2014-12-23T17:41:35.393+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a94e9a8f659f34bfb8e3 2014-12-23T17:41:35.393+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-12 based on current shard version 3|0||5499a94800df4929841903d4, current metadata version is 3|0||5499a94800df4929841903d4 2014-12-23T17:41:35.394+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-12 from shard version 3|0||5499a94800df4929841903d4 to shard version 3|1||5499a94800df4929841903d4 2014-12-23T17:41:35.394+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a94800df4929841903d4, took 0ms 2014-12-23T17:41:35.394+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a94800df4929841903d4 2014-12-23T17:41:35.699+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:35-5499a94f9a8f659f34bfb8e4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356495699), what: "split", ns: "benchdb1.COL-12", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a94800df4929841903d4') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a94800df4929841903d4') } } } 2014-12-23T17:41:36.077+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-12/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:36.077+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-12", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-12-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a94800df4929841903d4') } ntoreturn:1 keyUpdates:0 reslen:37 1286ms 2014-12-23T17:41:38.983+0000 I INDEX [conn9] build index on: benchdb1.COL-13 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-13" } 2014-12-23T17:41:38.983+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:41:38.984+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:39.286+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 1|2||5499a94f00df4929841903d8, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:39.286+0000 I SHARDING [conn2] collection benchdb1.COL-13 was previously unsharded, new metadata loaded with shard version 1|2||5499a94f00df4929841903d8 2014-12-23T17:41:39.286+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a94f00df4929841903d8, took 0ms 2014-12-23T17:41:39.327+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:39.327+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94f00df4929841903d8') } 2014-12-23T17:41:39.934+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9539a8f659f34bfb8e6 2014-12-23T17:41:39.934+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:39-5499a9539a8f659f34bfb8e7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356499934), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:40.003+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 1|2||5499a94f00df4929841903d8, current metadata version is 1|2||5499a94f00df4929841903d8 2014-12-23T17:41:40.004+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-13 already up to date (shard version : 1|2||5499a94f00df4929841903d8, took 0ms) 2014-12-23T17:41:40.004+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a94f00df4929841903d8 2014-12-23T17:41:40.008+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:40.011+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:40.013+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:40.017+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:40.017+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:40.018+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:40.018+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a94f00df4929841903d8 2014-12-23T17:41:40.073+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:40.073+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a94f00df4929841903d8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T17:41:40.379+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:40-5499a9549a8f659f34bfb8e8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356500379), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:40.432+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:40.442+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:40.442+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 17345689 2014-12-23T17:41:40.442+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:40.442+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:40.741+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:40.741+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:40-5499a9549a8f659f34bfb8e9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356500741), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 676, step 3 of 6: 5, step 4 of 6: 8, step 5 of 6: 424, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:40.802+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94f00df4929841903d8') } ntoreturn:1 keyUpdates:0 reslen:37 1475ms 2014-12-23T17:41:40.803+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:40.804+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94f00df4929841903d8') } 2014-12-23T17:41:41.411+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9549a8f659f34bfb8ea 2014-12-23T17:41:41.411+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:41-5499a9559a8f659f34bfb8eb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356501411), what: "moveChunk.start", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:41.473+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 2|0||5499a94f00df4929841903d8, current metadata version is 2|0||5499a94f00df4929841903d8 2014-12-23T17:41:41.474+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-13 from shard version 2|0||5499a94f00df4929841903d8 to shard version 2|1||5499a94f00df4929841903d8 2014-12-23T17:41:41.474+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a94f00df4929841903d8, took 0ms 2014-12-23T17:41:41.474+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a94f00df4929841903d8 2014-12-23T17:41:41.476+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:41.478+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:41.481+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:41.485+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:41.494+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:41.510+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:41.510+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:41.511+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:41.511+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a94f00df4929841903d8 2014-12-23T17:41:41.566+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:41.567+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a94f00df4929841903d8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-13' 2014-12-23T17:41:41.872+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:41-5499a9559a8f659f34bfb8ec", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356501872), what: "moveChunk.commit", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:41.935+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:41.947+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:41.947+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 17457732 2014-12-23T17:41:41.947+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-13 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:41.947+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:42.251+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:42.251+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:42-5499a9569a8f659f34bfb8ed", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356502251), what: "moveChunk.from", ns: "benchdb1.COL-13", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 2, step 4 of 6: 33, step 5 of 6: 437, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:42.311+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-13", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-13-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a94f00df4929841903d8') } ntoreturn:1 keyUpdates:0 reslen:37 1508ms 2014-12-23T17:41:42.313+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a94f00df4929841903d8') } 2014-12-23T17:41:42.794+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9569a8f659f34bfb8ee 2014-12-23T17:41:42.794+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-13 based on current shard version 3|0||5499a94f00df4929841903d8, current metadata version is 3|0||5499a94f00df4929841903d8 2014-12-23T17:41:42.795+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-13 from shard version 3|0||5499a94f00df4929841903d8 to shard version 3|1||5499a94f00df4929841903d8 2014-12-23T17:41:42.795+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a94f00df4929841903d8, took 0ms 2014-12-23T17:41:42.795+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a94f00df4929841903d8 2014-12-23T17:41:43.046+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:43-5499a9579a8f659f34bfb8ef", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356503046), what: "split", ns: "benchdb1.COL-13", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a94f00df4929841903d8') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a94f00df4929841903d8') } } } 2014-12-23T17:41:43.239+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-13/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:43.239+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-13", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-13-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a94f00df4929841903d8') } ntoreturn:1 keyUpdates:0 reslen:37 925ms 2014-12-23T17:41:43.454+0000 I QUERY [conn9] command benchdb1.$cmd command: insert { insert: "COL-51", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 115ms 2014-12-23T17:41:43.743+0000 I QUERY [conn9] command benchdb1.$cmd command: insert { insert: "COL-51", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 107ms 2014-12-23T17:41:46.011+0000 I INDEX [conn9] build index on: benchdb1.COL-14 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-14" } 2014-12-23T17:41:46.011+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:41:48.702+0000 I INDEX [conn9] build index done. scanned 0 total records. 2 secs 2014-12-23T17:41:48.702+0000 I WRITE [conn9] insert benchdb1.system.indexes query: { ns: "benchdb1.COL-14", key: { shardkey: "hashed" }, name: "shardkey_hashed" } ninserted:1 keyUpdates:0 3019ms 2014-12-23T17:41:48.702+0000 I QUERY [conn9] command benchdb1.$cmd command: insert { insert: "system.indexes", documents: [ { ns: "benchdb1.COL-14", key: { shardkey: "hashed" }, name: "shardkey_hashed" } ], ordered: true, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 3019ms 2014-12-23T17:41:48.702+0000 I WRITE [conn2] insert benchdb1.COL-51 query: { _id: ObjectId('5498b6880315362bd8591eb2'), shardkey: "nJ1hKa3XZAGQgro8UQ1Ql2Uc", payload: "XSl2HLy3XRE30kDIZuwZ3SGxDfgtj6KqxvKKtJRHlY84ndTm9xoLYK5usVUmTDJlf3VovAA5SQKuX75cDHqUEQkOVIxnwPhY4XGdDx7JoLSJTocaC0cUKvTqbgb8OYbZE6p4bXObq2LoAwpojAriJA..." } ninserted:1 keyUpdates:0 3015ms 2014-12-23T17:41:48.726+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-51", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 3039ms 2014-12-23T17:41:49.046+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 1|2||5499a95900df4929841903dd, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:49.047+0000 I SHARDING [conn2] collection benchdb1.COL-14 was previously unsharded, new metadata loaded with shard version 1|2||5499a95900df4929841903dd 2014-12-23T17:41:49.047+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a95900df4929841903dd, took 0ms 2014-12-23T17:41:49.102+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:49.103+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a95900df4929841903dd') } 2014-12-23T17:41:49.629+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a95d9a8f659f34bfb8f1 2014-12-23T17:41:49.629+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:49-5499a95d9a8f659f34bfb8f2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356509629), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:49.685+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 1|2||5499a95900df4929841903dd, current metadata version is 1|2||5499a95900df4929841903dd 2014-12-23T17:41:49.686+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-14 already up to date (shard version : 1|2||5499a95900df4929841903dd, took 0ms) 2014-12-23T17:41:49.686+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a95900df4929841903dd 2014-12-23T17:41:49.703+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:49.706+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:49.708+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:49.712+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:49.713+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:49.713+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:49.713+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a95900df4929841903dd 2014-12-23T17:41:49.755+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:49.755+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a95900df4929841903dd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T17:41:49.943+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:49-5499a95d9a8f659f34bfb8f3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356509943), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:50.004+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:50.008+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:50.008+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 17740503 2014-12-23T17:41:50.008+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:50.008+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:50.195+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:50.195+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:50-5499a95e9a8f659f34bfb8f4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356510195), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 583, step 3 of 6: 17, step 4 of 6: 8, step 5 of 6: 294, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:50.204+0000 I WRITE [conn9] insert benchdb1.COL-51 query: { _id: ObjectId('5498b68c0315362bd859aa15'), shardkey: "5dIafDiuQWMREkb8DE0YLlDm", payload: "APzRjqRpfie4I8wCgvxC658DVgbbRsoTVI6tLTMQD3zqNEZxrN5tdM3G4zW3ZHpOnOyIMMSjtTBlj95I4ikoZveVc7G3gv0dAMm5JN2sVwjhwHfl3W3SNaTiZgoz30xObYDGYTrf9LVaNqJ45Lu3R6..." } ninserted:1 keyUpdates:0 154ms 2014-12-23T17:41:50.212+0000 I QUERY [conn9] command benchdb1.$cmd command: insert { insert: "COL-51", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 176ms 2014-12-23T17:41:50.228+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a95900df4929841903dd') } ntoreturn:1 keyUpdates:0 reslen:37 1126ms 2014-12-23T17:41:50.229+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:50.230+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a95900df4929841903dd') } 2014-12-23T17:41:50.742+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a95e9a8f659f34bfb8f5 2014-12-23T17:41:50.742+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:50-5499a95e9a8f659f34bfb8f6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356510742), what: "moveChunk.start", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:50.766+0000 I SHARDING [conn3] remotely refreshing metadata for benchdb1.COL-5 based on current shard version 3|3||5499a91a00df4929841903b6, current metadata version is 3|3||5499a91a00df4929841903b6 2014-12-23T17:41:50.767+0000 I SHARDING [conn3] updating metadata for benchdb1.COL-5 from shard version 3|3||5499a91a00df4929841903b6 to shard version 3|3||5499a91a00df4929841903b6 2014-12-23T17:41:50.768+0000 I SHARDING [conn3] collection version was loaded at version 4|1||5499a91a00df4929841903b6, took 1ms 2014-12-23T17:41:50.768+0000 I SHARDING [migrateThread] starting receiving-end of migration of chunk { shardkey: -1571812759196788008 } -> { shardkey: -64133374566864836 } for collection benchdb1.COL-5 from ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018 at epoch 5499a91a00df4929841903b6 2014-12-23T17:41:50.775+0000 I SHARDING [migrateThread] Deleter starting delete for: benchdb1.COL-5 from { shardkey: -1571812759196788008 } -> { shardkey: -64133374566864836 }, with opId: 17847714 2014-12-23T17:41:50.776+0000 I SHARDING [migrateThread] rangeDeleter deleted 0 documents for benchdb1.COL-5 from { shardkey: -1571812759196788008 } -> { shardkey: -64133374566864836 } 2014-12-23T17:41:50.780+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 2|0||5499a95900df4929841903dd, current metadata version is 2|0||5499a95900df4929841903dd 2014-12-23T17:41:50.781+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-14 from shard version 2|0||5499a95900df4929841903dd to shard version 2|1||5499a95900df4929841903dd 2014-12-23T17:41:50.781+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a95900df4929841903dd, took 0ms 2014-12-23T17:41:50.781+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a95900df4929841903dd 2014-12-23T17:41:50.796+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:50.799+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:50.801+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:50.806+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:50.815+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:50.831+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:50.831+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:50.832+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:50.832+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a95900df4929841903dd 2014-12-23T17:41:50.893+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:50.894+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a95900df4929841903dd through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-14' 2014-12-23T17:41:51.201+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:51-5499a95f9a8f659f34bfb8f7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356511200), what: "moveChunk.commit", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:51.265+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:51.279+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:51.279+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 17808043 2014-12-23T17:41:51.279+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-14 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:51.279+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:51.418+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:41:50 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:41:51.569+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:51.569+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:51-5499a95f9a8f659f34bfb8f8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356511569), what: "moveChunk.from", ns: "benchdb1.COL-14", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 551, step 3 of 6: 16, step 4 of 6: 34, step 5 of 6: 447, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:51.629+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-14", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-14-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a95900df4929841903dd') } ntoreturn:1 keyUpdates:0 reslen:37 1399ms 2014-12-23T17:41:51.630+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a95900df4929841903dd') } 2014-12-23T17:41:52.235+0000 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section 2014-12-23T17:41:52.235+0000 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'benchdb1.COL-5' { shardkey: -1571812759196788008 } -> { shardkey: -64133374566864836 } 2014-12-23T17:41:52.239+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a95f9a8f659f34bfb8f9 2014-12-23T17:41:52.239+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-14 based on current shard version 3|0||5499a95900df4929841903dd, current metadata version is 3|0||5499a95900df4929841903dd 2014-12-23T17:41:52.248+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-14 from shard version 3|0||5499a95900df4929841903dd to shard version 3|1||5499a95900df4929841903dd 2014-12-23T17:41:52.248+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a95900df4929841903dd, took 0ms 2014-12-23T17:41:52.248+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a95900df4929841903dd 2014-12-23T17:41:52.553+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:52-5499a9609a8f659f34bfb8fa", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356512553), what: "split", ns: "benchdb1.COL-14", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a95900df4929841903dd') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a95900df4929841903dd') } } } 2014-12-23T17:41:52.826+0000 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'benchdb1.COL-5' { shardkey: -1571812759196788008 } -> { shardkey: -64133374566864836 } 2014-12-23T17:41:52.840+0000 I SHARDING [migrateThread] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:52-5499a9609a8f659f34bfb8fb", server: "ip-10-238-44-130", clientAddr: ":27017", time: new Date(1419356512840), what: "moveChunk.to", ns: "benchdb1.COL-5", details: { min: { shardkey: -1571812759196788008 }, max: { shardkey: -64133374566864836 }, step 1 of 5: 7, step 2 of 5: 0, step 3 of 5: 1459, step 4 of 5: 0, step 5 of 5: 605, note: "success" } } 2014-12-23T17:41:52.923+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-14/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:52.923+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-14", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-14-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a95900df4929841903dd') } ntoreturn:1 keyUpdates:0 reslen:37 1292ms 2014-12-23T17:41:53.122+0000 I QUERY [conn7] command admin.$cmd command: _recvChunkCommit { _recvChunkCommit: 1 } ntoreturn:1 keyUpdates:0 reslen:333 300ms 2014-12-23T17:41:55.198+0000 I INDEX [conn2] build index on: benchdb1.COL-15 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-15" } 2014-12-23T17:41:55.198+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:41:55.199+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:41:55.556+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 1|2||5499a96000df4929841903e3, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:41:55.557+0000 I SHARDING [conn2] collection benchdb1.COL-15 was previously unsharded, new metadata loaded with shard version 1|2||5499a96000df4929841903e3 2014-12-23T17:41:55.557+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a96000df4929841903e3, took 0ms 2014-12-23T17:41:55.628+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:55.628+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96000df4929841903e3') } 2014-12-23T17:41:55.953+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9639a8f659f34bfb8fd 2014-12-23T17:41:55.953+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:55-5499a9639a8f659f34bfb8fe", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356515953), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:41:56.036+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 1|2||5499a96000df4929841903e3, current metadata version is 1|2||5499a96000df4929841903e3 2014-12-23T17:41:56.036+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-15 already up to date (shard version : 1|2||5499a96000df4929841903e3, took 0ms) 2014-12-23T17:41:56.037+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a96000df4929841903e3 2014-12-23T17:41:56.037+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:56.040+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:56.042+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:56.047+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:56.047+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:56.047+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:56.047+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a96000df4929841903e3 2014-12-23T17:41:56.096+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:56.096+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a96000df4929841903e3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T17:41:56.195+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a96000df4929841903e3') } 2014-12-23T17:41:56.195+0000 W SHARDING [conn4] migration already in progress 2014-12-23T17:41:56.195+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:56-5499a9649a8f659f34bfb8ff", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356516195), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:41:56.292+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:56-5499a9649a8f659f34bfb900", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356516292), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:56.343+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:56.343+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:56.343+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 18105960 2014-12-23T17:41:56.343+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:41:56.343+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:56.548+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:56.548+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:56-5499a9649a8f659f34bfb901", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356516548), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 408, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 296, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:41:56.605+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96000df4929841903e3') } ntoreturn:1 keyUpdates:0 reslen:37 977ms 2014-12-23T17:41:56.606+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:41:56.606+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96000df4929841903e3') } 2014-12-23T17:41:57.213+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9649a8f659f34bfb902 2014-12-23T17:41:57.214+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:57-5499a9659a8f659f34bfb903", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356517214), what: "moveChunk.start", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:41:57.277+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 2|0||5499a96000df4929841903e3, current metadata version is 2|0||5499a96000df4929841903e3 2014-12-23T17:41:57.277+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-15 from shard version 2|0||5499a96000df4929841903e3 to shard version 2|1||5499a96000df4929841903e3 2014-12-23T17:41:57.278+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a96000df4929841903e3, took 0ms 2014-12-23T17:41:57.278+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a96000df4929841903e3 2014-12-23T17:41:57.282+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:41:57.284+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:57.287+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:57.291+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:57.300+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:57.316+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:41:57.317+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:41:57.317+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:41:57.317+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a96000df4929841903e3 2014-12-23T17:41:57.369+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:41:57.369+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a96000df4929841903e3 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-15' 2014-12-23T17:41:57.682+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:57-5499a9659a8f659f34bfb904", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356517682), what: "moveChunk.commit", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:41:57.758+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:57.758+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:41:57.758+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 18165438 2014-12-23T17:41:57.758+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-15 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:41:57.758+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:41:58.062+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:58.062+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:58-5499a9669a8f659f34bfb905", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356518062), what: "moveChunk.from", ns: "benchdb1.COL-15", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 671, step 3 of 6: 5, step 4 of 6: 33, step 5 of 6: 441, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:41:58.328+0000 I WRITE [conn9] insert benchdb1.COL-53 query: { _id: ObjectId('5498b6a50315362bd85d1fec'), shardkey: "uwdljxLVTlNuckBaYCuPy6Rf", payload: "Er8p60DqJeusLadaiJr3m71apJvztf3L1YDH5TyMoJvBa656qsMioaZ9H6WsHBD5YqfEsLOXofJwBEH0NLQqXDK1ytJKcWHpvUSnN4DO2b6vaxzDKBMASf7aqTaQuNR1rA99BXpzICD9V4nGcxHmly..." } ninserted:1 keyUpdates:0 248ms 2014-12-23T17:41:58.335+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-15", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-15-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96000df4929841903e3') } ntoreturn:1 keyUpdates:0 reslen:37 1728ms 2014-12-23T17:41:58.337+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a96000df4929841903e3') } 2014-12-23T17:41:58.347+0000 I QUERY [conn9] command benchdb1.$cmd command: insert { insert: "COL-53", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 268ms 2014-12-23T17:41:58.944+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9669a8f659f34bfb906 2014-12-23T17:41:58.944+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-15 based on current shard version 3|0||5499a96000df4929841903e3, current metadata version is 3|0||5499a96000df4929841903e3 2014-12-23T17:41:58.945+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-15 from shard version 3|0||5499a96000df4929841903e3 to shard version 3|1||5499a96000df4929841903e3 2014-12-23T17:41:58.945+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a96000df4929841903e3, took 0ms 2014-12-23T17:41:58.945+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a96000df4929841903e3 2014-12-23T17:41:59.251+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:41:59-5499a9679a8f659f34bfb907", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356519251), what: "split", ns: "benchdb1.COL-15", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a96000df4929841903e3') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a96000df4929841903e3') } } } 2014-12-23T17:41:59.650+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-15/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:41:59.650+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-15", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-15-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a96000df4929841903e3') } ntoreturn:1 keyUpdates:0 reslen:37 1313ms 2014-12-23T17:42:02.316+0000 I INDEX [conn9] build index on: benchdb1.COL-16 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-16" } 2014-12-23T17:42:02.316+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:42:02.317+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:02.629+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 1|2||5499a96700df4929841903e7, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:02.629+0000 I SHARDING [conn2] collection benchdb1.COL-16 was previously unsharded, new metadata loaded with shard version 1|2||5499a96700df4929841903e7 2014-12-23T17:42:02.629+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a96700df4929841903e7, took 0ms 2014-12-23T17:42:02.675+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:02.676+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96700df4929841903e7') } 2014-12-23T17:42:03.285+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a96a9a8f659f34bfb909 2014-12-23T17:42:03.285+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:03-5499a96b9a8f659f34bfb90a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356523285), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:03.643+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 1|2||5499a96700df4929841903e7, current metadata version is 1|2||5499a96700df4929841903e7 2014-12-23T17:42:03.644+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-16 already up to date (shard version : 1|2||5499a96700df4929841903e7, took 0ms) 2014-12-23T17:42:03.644+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a96700df4929841903e7 2014-12-23T17:42:03.656+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:03.659+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:03.662+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:03.666+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:03.675+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:03.675+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:03.675+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:03.676+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a96700df4929841903e7 2014-12-23T17:42:03.727+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:03.727+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a96700df4929841903e7 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T17:42:04.033+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:04-5499a96c9a8f659f34bfb90b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356524033), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:04.088+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:04.098+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:04.099+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 18573862 2014-12-23T17:42:04.099+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:04.099+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:04.402+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:04.403+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:04-5499a96c9a8f659f34bfb90c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356524403), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 968, step 3 of 6: 13, step 4 of 6: 17, step 5 of 6: 423, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:04.462+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96700df4929841903e7') } ntoreturn:1 keyUpdates:0 reslen:37 1787ms 2014-12-23T17:42:04.464+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:04.464+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96700df4929841903e7') } 2014-12-23T17:42:05.072+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a96c9a8f659f34bfb90d 2014-12-23T17:42:05.072+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:05-5499a96d9a8f659f34bfb90e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356525072), what: "moveChunk.start", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:05.136+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 2|0||5499a96700df4929841903e7, current metadata version is 2|0||5499a96700df4929841903e7 2014-12-23T17:42:05.137+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-16 from shard version 2|0||5499a96700df4929841903e7 to shard version 2|1||5499a96700df4929841903e7 2014-12-23T17:42:05.137+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a96700df4929841903e7, took 0ms 2014-12-23T17:42:05.138+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a96700df4929841903e7 2014-12-23T17:42:05.144+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:05.147+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:05.149+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:05.154+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:05.162+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:05.179+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:05.179+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:05.179+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:05.180+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a96700df4929841903e7 2014-12-23T17:42:05.237+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:05.237+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a96700df4929841903e7 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-16' 2014-12-23T17:42:05.542+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:05-5499a96d9a8f659f34bfb90f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356525542), what: "moveChunk.commit", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:05.615+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:05.621+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:05.621+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 18687633 2014-12-23T17:42:05.622+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-16 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:05.622+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:05.925+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:05.925+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:05-5499a96d9a8f659f34bfb910", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356525925), what: "moveChunk.from", ns: "benchdb1.COL-16", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 673, step 3 of 6: 7, step 4 of 6: 33, step 5 of 6: 442, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:05.981+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-16", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-16-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96700df4929841903e7') } ntoreturn:1 keyUpdates:0 reslen:37 1516ms 2014-12-23T17:42:05.982+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a96700df4929841903e7') } 2014-12-23T17:42:06.550+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a96d9a8f659f34bfb911 2014-12-23T17:42:06.550+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 based on current shard version 3|0||5499a96700df4929841903e7, current metadata version is 3|0||5499a96700df4929841903e7 2014-12-23T17:42:06.551+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-16 from shard version 3|0||5499a96700df4929841903e7 to shard version 3|1||5499a96700df4929841903e7 2014-12-23T17:42:06.551+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a96700df4929841903e7, took 0ms 2014-12-23T17:42:06.551+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a96700df4929841903e7 2014-12-23T17:42:06.807+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:06-5499a96e9a8f659f34bfb912", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356526807), what: "split", ns: "benchdb1.COL-16", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a96700df4929841903e7') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a96700df4929841903e7') } } } 2014-12-23T17:42:07.149+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-16/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:07.149+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-16", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-16-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a96700df4929841903e7') } ntoreturn:1 keyUpdates:0 reslen:37 1167ms 2014-12-23T17:42:09.333+0000 I INDEX [conn2] build index on: benchdb1.COL-17 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-17" } 2014-12-23T17:42:09.333+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:42:09.334+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:09.648+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-17 with requested shard version 1|2||5499a96e00df4929841903ec, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:09.649+0000 I SHARDING [conn2] collection benchdb1.COL-17 was previously unsharded, new metadata loaded with shard version 1|2||5499a96e00df4929841903ec 2014-12-23T17:42:09.649+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a96e00df4929841903ec, took 0ms 2014-12-23T17:42:09.695+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:09.696+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96e00df4929841903ec') } 2014-12-23T17:42:10.304+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9719a8f659f34bfb914 2014-12-23T17:42:10.304+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:10-5499a9729a8f659f34bfb915", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356530304), what: "moveChunk.start", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:10.379+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 1|2||5499a96e00df4929841903ec, current metadata version is 1|2||5499a96e00df4929841903ec 2014-12-23T17:42:10.380+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-17 already up to date (shard version : 1|2||5499a96e00df4929841903ec, took 0ms) 2014-12-23T17:42:10.380+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a96e00df4929841903ec 2014-12-23T17:42:10.381+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:10.384+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:10.386+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:10.391+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:10.391+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:10.391+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:10.391+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a96e00df4929841903ec 2014-12-23T17:42:10.450+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:10.450+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a96e00df4929841903ec through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-17' 2014-12-23T17:42:10.742+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:10-5499a9729a8f659f34bfb916", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356530742), what: "moveChunk.commit", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:10.798+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:10.806+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:10.806+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-17 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 19003677 2014-12-23T17:42:10.806+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-17 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:10.806+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:11.110+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:11.110+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:11-5499a9739a8f659f34bfb917", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356531110), what: "moveChunk.from", ns: "benchdb1.COL-17", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 684, step 3 of 6: 2, step 4 of 6: 8, step 5 of 6: 415, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:11.165+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96e00df4929841903ec') } ntoreturn:1 keyUpdates:0 reslen:37 1470ms 2014-12-23T17:42:11.167+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:11.168+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96e00df4929841903ec') } 2014-12-23T17:42:11.775+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9739a8f659f34bfb918 2014-12-23T17:42:11.775+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:11-5499a9739a8f659f34bfb919", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356531775), what: "moveChunk.start", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:11.837+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 2|0||5499a96e00df4929841903ec, current metadata version is 2|0||5499a96e00df4929841903ec 2014-12-23T17:42:11.838+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-17 from shard version 2|0||5499a96e00df4929841903ec to shard version 2|1||5499a96e00df4929841903ec 2014-12-23T17:42:11.838+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a96e00df4929841903ec, took 0ms 2014-12-23T17:42:11.838+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a96e00df4929841903ec 2014-12-23T17:42:11.846+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:11.849+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:11.851+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:11.856+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:11.864+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:11.881+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:11.881+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:11.881+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:11.881+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a96e00df4929841903ec 2014-12-23T17:42:11.934+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:11.934+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a96e00df4929841903ec through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-17' 2014-12-23T17:42:12.240+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:12-5499a9749a8f659f34bfb91a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356532240), what: "moveChunk.commit", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:12.309+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:12.309+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:12.310+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-17 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 19099197 2014-12-23T17:42:12.310+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-17 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:12.310+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:12.614+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:12.614+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:12-5499a9749a8f659f34bfb91b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356532614), what: "moveChunk.from", ns: "benchdb1.COL-17", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 8, step 4 of 6: 33, step 5 of 6: 428, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:12.666+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-17", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-17-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a96e00df4929841903ec') } ntoreturn:1 keyUpdates:0 reslen:37 1498ms 2014-12-23T17:42:12.669+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-17", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-17-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a96e00df4929841903ec') } 2014-12-23T17:42:13.276+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9749a8f659f34bfb91c 2014-12-23T17:42:13.276+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-17 based on current shard version 3|0||5499a96e00df4929841903ec, current metadata version is 3|0||5499a96e00df4929841903ec 2014-12-23T17:42:13.277+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-17 from shard version 3|0||5499a96e00df4929841903ec to shard version 3|1||5499a96e00df4929841903ec 2014-12-23T17:42:13.277+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a96e00df4929841903ec, took 0ms 2014-12-23T17:42:13.277+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a96e00df4929841903ec 2014-12-23T17:42:13.582+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:13-5499a9759a8f659f34bfb91d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356533582), what: "split", ns: "benchdb1.COL-17", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a96e00df4929841903ec') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a96e00df4929841903ec') } } } 2014-12-23T17:42:14.217+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-17/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:14.217+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-17", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-17-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a96e00df4929841903ec') } ntoreturn:1 keyUpdates:0 reslen:37 1547ms 2014-12-23T17:42:16.875+0000 I INDEX [conn2] build index on: benchdb1.COL-18 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-18" } 2014-12-23T17:42:16.875+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:42:16.876+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:17.180+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-18 with requested shard version 1|2||5499a97500df4929841903ef, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:17.181+0000 I SHARDING [conn9] collection benchdb1.COL-18 was previously unsharded, new metadata loaded with shard version 1|2||5499a97500df4929841903ef 2014-12-23T17:42:17.181+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a97500df4929841903ef, took 0ms 2014-12-23T17:42:17.224+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:17.225+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97500df4929841903ef') } 2014-12-23T17:42:17.832+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9799a8f659f34bfb91f 2014-12-23T17:42:17.832+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:17-5499a9799a8f659f34bfb920", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356537832), what: "moveChunk.start", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:17.896+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 1|2||5499a97500df4929841903ef, current metadata version is 1|2||5499a97500df4929841903ef 2014-12-23T17:42:17.897+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-18 already up to date (shard version : 1|2||5499a97500df4929841903ef, took 0ms) 2014-12-23T17:42:17.897+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a97500df4929841903ef 2014-12-23T17:42:17.907+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:17.910+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:17.912+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:17.917+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:17.917+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:17.917+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:17.917+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a97500df4929841903ef 2014-12-23T17:42:17.974+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:17.974+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a97500df4929841903ef through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-18' 2014-12-23T17:42:18.239+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:18-5499a97a9a8f659f34bfb921", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356538239), what: "moveChunk.commit", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:18.311+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:18.311+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:18.311+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-18 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 19468322 2014-12-23T17:42:18.311+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-18 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:18.311+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:18.574+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:18.574+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:18-5499a97a9a8f659f34bfb922", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356538574), what: "moveChunk.from", ns: "benchdb1.COL-18", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 672, step 3 of 6: 11, step 4 of 6: 8, step 5 of 6: 394, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:18.613+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97500df4929841903ef') } ntoreturn:1 keyUpdates:0 reslen:37 1389ms 2014-12-23T17:42:18.615+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:18.615+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97500df4929841903ef') } 2014-12-23T17:42:18.900+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a97a9a8f659f34bfb923 2014-12-23T17:42:18.900+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:18-5499a97a9a8f659f34bfb924", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356538900), what: "moveChunk.start", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:18.949+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 2|0||5499a97500df4929841903ef, current metadata version is 2|0||5499a97500df4929841903ef 2014-12-23T17:42:18.950+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-18 from shard version 2|0||5499a97500df4929841903ef to shard version 2|1||5499a97500df4929841903ef 2014-12-23T17:42:18.950+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a97500df4929841903ef, took 0ms 2014-12-23T17:42:18.950+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a97500df4929841903ef 2014-12-23T17:42:18.951+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:18.955+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:18.958+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:18.962+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:18.971+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:18.987+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:19.005+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:19.006+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:19.013+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a97500df4929841903ef 2014-12-23T17:42:19.171+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:19.171+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a97500df4929841903ef through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-18' 2014-12-23T17:42:19.333+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:19-5499a97b9a8f659f34bfb925", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356539333), what: "moveChunk.commit", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:19.386+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:19.386+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:19.386+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-18 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 19558938 2014-12-23T17:42:19.386+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-18 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:19.386+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:19.569+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:19.569+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:19-5499a97b9a8f659f34bfb926", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356539569), what: "moveChunk.from", ns: "benchdb1.COL-18", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 335, step 3 of 6: 1, step 4 of 6: 53, step 5 of 6: 380, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:19.635+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-18", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-18-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97500df4929841903ef') } ntoreturn:1 keyUpdates:0 reslen:37 1020ms 2014-12-23T17:42:19.637+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-18", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-18-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a97500df4929841903ef') } 2014-12-23T17:42:20.196+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a97b9a8f659f34bfb927 2014-12-23T17:42:20.196+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-18 based on current shard version 3|0||5499a97500df4929841903ef, current metadata version is 3|0||5499a97500df4929841903ef 2014-12-23T17:42:20.197+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-18 from shard version 3|0||5499a97500df4929841903ef to shard version 3|1||5499a97500df4929841903ef 2014-12-23T17:42:20.197+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a97500df4929841903ef, took 0ms 2014-12-23T17:42:20.197+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a97500df4929841903ef 2014-12-23T17:42:20.502+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:20-5499a97c9a8f659f34bfb928", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356540502), what: "split", ns: "benchdb1.COL-18", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a97500df4929841903ef') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a97500df4929841903ef') } } } 2014-12-23T17:42:20.886+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-18/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:20.886+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-18", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-18-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a97500df4929841903ef') } ntoreturn:1 keyUpdates:0 reslen:37 1248ms 2014-12-23T17:42:21.917+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:42:21 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:42:23.466+0000 I INDEX [conn2] build index on: benchdb1.COL-19 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-19" } 2014-12-23T17:42:23.466+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:42:23.467+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:23.765+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-19 with requested shard version 1|2||5499a97c00df4929841903f4, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:23.766+0000 I SHARDING [conn9] collection benchdb1.COL-19 was previously unsharded, new metadata loaded with shard version 1|2||5499a97c00df4929841903f4 2014-12-23T17:42:23.766+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a97c00df4929841903f4, took 0ms 2014-12-23T17:42:23.808+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:23.809+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97c00df4929841903f4') } 2014-12-23T17:42:24.369+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a97f9a8f659f34bfb92a 2014-12-23T17:42:24.369+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:24-5499a9809a8f659f34bfb92b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356544369), what: "moveChunk.start", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:24.715+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 1|2||5499a97c00df4929841903f4, current metadata version is 1|2||5499a97c00df4929841903f4 2014-12-23T17:42:24.716+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-19 already up to date (shard version : 1|2||5499a97c00df4929841903f4, took 0ms) 2014-12-23T17:42:24.716+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a97c00df4929841903f4 2014-12-23T17:42:24.725+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:24.727+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:24.730+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:24.734+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:24.734+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:24.735+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:24.735+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a97c00df4929841903f4 2014-12-23T17:42:24.792+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:24.792+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a97c00df4929841903f4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-19' 2014-12-23T17:42:25.097+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:25-5499a9819a8f659f34bfb92c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356545097), what: "moveChunk.commit", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:25.153+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:25.162+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:25.162+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-19 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 19868230 2014-12-23T17:42:25.162+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-19 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:25.162+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:25.465+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:25.465+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:25-5499a9819a8f659f34bfb92d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356545465), what: "moveChunk.from", ns: "benchdb1.COL-19", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 907, step 3 of 6: 9, step 4 of 6: 8, step 5 of 6: 427, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:25.519+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97c00df4929841903f4') } ntoreturn:1 keyUpdates:0 reslen:37 1711ms 2014-12-23T17:42:25.521+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:25.521+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97c00df4929841903f4') } 2014-12-23T17:42:26.128+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9819a8f659f34bfb92e 2014-12-23T17:42:26.128+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:26-5499a9829a8f659f34bfb92f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356546128), what: "moveChunk.start", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:26.206+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 2|0||5499a97c00df4929841903f4, current metadata version is 2|0||5499a97c00df4929841903f4 2014-12-23T17:42:26.206+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-19 from shard version 2|0||5499a97c00df4929841903f4 to shard version 2|1||5499a97c00df4929841903f4 2014-12-23T17:42:26.206+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a97c00df4929841903f4, took 0ms 2014-12-23T17:42:26.206+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a97c00df4929841903f4 2014-12-23T17:42:26.221+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:26.224+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:26.226+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:26.230+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:26.239+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:26.255+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:26.255+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:26.256+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:26.256+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a97c00df4929841903f4 2014-12-23T17:42:26.305+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:26.305+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a97c00df4929841903f4 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-19' 2014-12-23T17:42:26.610+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:26-5499a9829a8f659f34bfb930", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356546610), what: "moveChunk.commit", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:26.674+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:26.676+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:26.676+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-19 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 19973402 2014-12-23T17:42:26.676+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-19 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:26.676+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:26.979+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:26.979+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:26-5499a9829a8f659f34bfb931", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356546979), what: "moveChunk.from", ns: "benchdb1.COL-19", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 685, step 3 of 6: 15, step 4 of 6: 33, step 5 of 6: 420, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:27.043+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-19", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-19-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a97c00df4929841903f4') } ntoreturn:1 keyUpdates:0 reslen:37 1521ms 2014-12-23T17:42:27.045+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-19", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-19-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a97c00df4929841903f4') } 2014-12-23T17:42:27.652+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9839a8f659f34bfb932 2014-12-23T17:42:27.652+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-19 based on current shard version 3|0||5499a97c00df4929841903f4, current metadata version is 3|0||5499a97c00df4929841903f4 2014-12-23T17:42:27.653+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-19 from shard version 3|0||5499a97c00df4929841903f4 to shard version 3|1||5499a97c00df4929841903f4 2014-12-23T17:42:27.653+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a97c00df4929841903f4, took 0ms 2014-12-23T17:42:27.653+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a97c00df4929841903f4 2014-12-23T17:42:27.958+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:27-5499a9839a8f659f34bfb933", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356547958), what: "split", ns: "benchdb1.COL-19", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a97c00df4929841903f4') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a97c00df4929841903f4') } } } 2014-12-23T17:42:28.352+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-19/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:28.352+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-19", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-19-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a97c00df4929841903f4') } ntoreturn:1 keyUpdates:0 reslen:37 1307ms 2014-12-23T17:42:30.865+0000 I INDEX [conn2] build index on: benchdb1.COL-20 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-20" } 2014-12-23T17:42:30.865+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:42:30.866+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:31.160+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-20 with requested shard version 1|2||5499a98300df4929841903f8, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:31.161+0000 I SHARDING [conn9] collection benchdb1.COL-20 was previously unsharded, new metadata loaded with shard version 1|2||5499a98300df4929841903f8 2014-12-23T17:42:31.161+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a98300df4929841903f8, took 0ms 2014-12-23T17:42:31.207+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:31.207+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98300df4929841903f8') } 2014-12-23T17:42:31.550+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9879a8f659f34bfb935 2014-12-23T17:42:31.550+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:31-5499a9879a8f659f34bfb936", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356551550), what: "moveChunk.start", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:31.600+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 1|2||5499a98300df4929841903f8, current metadata version is 1|2||5499a98300df4929841903f8 2014-12-23T17:42:31.601+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-20 already up to date (shard version : 1|2||5499a98300df4929841903f8, took 0ms) 2014-12-23T17:42:31.601+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a98300df4929841903f8 2014-12-23T17:42:31.602+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:31.605+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:31.607+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:31.611+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:31.611+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:31.612+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:31.612+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a98300df4929841903f8 2014-12-23T17:42:31.673+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:31.673+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a98300df4929841903f8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-20' 2014-12-23T17:42:31.978+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:31-5499a9879a8f659f34bfb937", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356551978), what: "moveChunk.commit", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:32.043+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:32.047+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:32.047+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-20 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 20315929 2014-12-23T17:42:32.047+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-20 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:32.047+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:32.351+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:32.351+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:32-5499a9889a8f659f34bfb938", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356552351), what: "moveChunk.from", ns: "benchdb1.COL-20", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 394, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 435, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:32.408+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98300df4929841903f8') } ntoreturn:1 keyUpdates:0 reslen:37 1201ms 2014-12-23T17:42:32.410+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:32.411+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98300df4929841903f8') } 2014-12-23T17:42:33.018+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9889a8f659f34bfb939 2014-12-23T17:42:33.018+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:33-5499a9899a8f659f34bfb93a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356553018), what: "moveChunk.start", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:33.096+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 2|0||5499a98300df4929841903f8, current metadata version is 2|0||5499a98300df4929841903f8 2014-12-23T17:42:33.097+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-20 from shard version 2|0||5499a98300df4929841903f8 to shard version 2|1||5499a98300df4929841903f8 2014-12-23T17:42:33.097+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a98300df4929841903f8, took 0ms 2014-12-23T17:42:33.097+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a98300df4929841903f8 2014-12-23T17:42:33.107+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:33.110+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:33.112+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:33.117+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:33.125+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:33.142+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:33.142+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:33.142+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:33.142+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a98300df4929841903f8 2014-12-23T17:42:33.197+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:33.197+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a98300df4929841903f8 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-20' 2014-12-23T17:42:33.468+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:33-5499a9899a8f659f34bfb93b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356553468), what: "moveChunk.commit", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:33.536+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:33.545+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:33.545+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-20 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 20389606 2014-12-23T17:42:33.545+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-20 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:33.546+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:33.849+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:33.849+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:33-5499a9899a8f659f34bfb93c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356553849), what: "moveChunk.from", ns: "benchdb1.COL-20", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 686, step 3 of 6: 11, step 4 of 6: 33, step 5 of 6: 403, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:33.903+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-20", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-20-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98300df4929841903f8') } ntoreturn:1 keyUpdates:0 reslen:37 1492ms 2014-12-23T17:42:33.904+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-20", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-20-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a98300df4929841903f8') } 2014-12-23T17:42:34.511+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9899a8f659f34bfb93d 2014-12-23T17:42:34.511+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 based on current shard version 3|0||5499a98300df4929841903f8, current metadata version is 3|0||5499a98300df4929841903f8 2014-12-23T17:42:34.512+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-20 from shard version 3|0||5499a98300df4929841903f8 to shard version 3|1||5499a98300df4929841903f8 2014-12-23T17:42:34.512+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a98300df4929841903f8, took 0ms 2014-12-23T17:42:34.513+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a98300df4929841903f8 2014-12-23T17:42:34.817+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:34-5499a98a9a8f659f34bfb93e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356554817), what: "split", ns: "benchdb1.COL-20", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a98300df4929841903f8') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a98300df4929841903f8') } } } 2014-12-23T17:42:35.470+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-20/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:35.470+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-20", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-20-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a98300df4929841903f8') } ntoreturn:1 keyUpdates:0 reslen:37 1565ms 2014-12-23T17:42:38.169+0000 I INDEX [conn2] build index on: benchdb1.COL-21 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-21" } 2014-12-23T17:42:38.169+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:42:38.170+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:38.507+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-21 with requested shard version 1|2||5499a98b00df4929841903fc, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:38.508+0000 I SHARDING [conn9] collection benchdb1.COL-21 was previously unsharded, new metadata loaded with shard version 1|2||5499a98b00df4929841903fc 2014-12-23T17:42:38.508+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a98b00df4929841903fc, took 0ms 2014-12-23T17:42:38.552+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:38.552+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98b00df4929841903fc') } 2014-12-23T17:42:39.160+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a98e9a8f659f34bfb940 2014-12-23T17:42:39.160+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:39-5499a98f9a8f659f34bfb941", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356559160), what: "moveChunk.start", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:39.229+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 1|2||5499a98b00df4929841903fc, current metadata version is 1|2||5499a98b00df4929841903fc 2014-12-23T17:42:39.230+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-21 already up to date (shard version : 1|2||5499a98b00df4929841903fc, took 0ms) 2014-12-23T17:42:39.230+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a98b00df4929841903fc 2014-12-23T17:42:39.232+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:39.235+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:39.237+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:39.242+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:39.242+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:39.242+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:39.242+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a98b00df4929841903fc 2014-12-23T17:42:39.295+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:39.295+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a98b00df4929841903fc through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-21' 2014-12-23T17:42:39.600+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:39-5499a98f9a8f659f34bfb942", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356559600), what: "moveChunk.commit", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:39.656+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:39.656+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:39.657+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-21 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 20767928 2014-12-23T17:42:39.657+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-21 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:39.657+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:39.960+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:39.960+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:39-5499a98f9a8f659f34bfb943", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356559960), what: "moveChunk.from", ns: "benchdb1.COL-21", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 677, step 3 of 6: 3, step 4 of 6: 8, step 5 of 6: 414, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:40.014+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98b00df4929841903fc') } ntoreturn:1 keyUpdates:0 reslen:37 1462ms 2014-12-23T17:42:40.016+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:40.016+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98b00df4929841903fc') } 2014-12-23T17:42:40.623+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9909a8f659f34bfb944 2014-12-23T17:42:40.623+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:40-5499a9909a8f659f34bfb945", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356560623), what: "moveChunk.start", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:40.693+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 2|0||5499a98b00df4929841903fc, current metadata version is 2|0||5499a98b00df4929841903fc 2014-12-23T17:42:40.694+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-21 from shard version 2|0||5499a98b00df4929841903fc to shard version 2|1||5499a98b00df4929841903fc 2014-12-23T17:42:40.695+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a98b00df4929841903fc, took 0ms 2014-12-23T17:42:40.695+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a98b00df4929841903fc 2014-12-23T17:42:40.698+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:40.701+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:40.703+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:40.708+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:40.716+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:40.732+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:40.732+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:40.733+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:40.733+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a98b00df4929841903fc 2014-12-23T17:42:40.809+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:40.809+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a98b00df4929841903fc through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-21' 2014-12-23T17:42:41.115+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:41-5499a9919a8f659f34bfb946", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356561115), what: "moveChunk.commit", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:41.403+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:41.406+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:41.406+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-21 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 20861982 2014-12-23T17:42:41.406+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-21 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:41.406+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:41.709+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:41.709+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:41-5499a9919a8f659f34bfb947", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356561709), what: "moveChunk.from", ns: "benchdb1.COL-21", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 678, step 3 of 6: 4, step 4 of 6: 33, step 5 of 6: 673, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:41.765+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-21", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-21-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a98b00df4929841903fc') } ntoreturn:1 keyUpdates:0 reslen:37 1749ms 2014-12-23T17:42:41.767+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-21", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-21-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a98b00df4929841903fc') } 2014-12-23T17:42:42.237+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9919a8f659f34bfb948 2014-12-23T17:42:42.237+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 based on current shard version 3|0||5499a98b00df4929841903fc, current metadata version is 3|0||5499a98b00df4929841903fc 2014-12-23T17:42:42.237+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-21 from shard version 3|0||5499a98b00df4929841903fc to shard version 3|1||5499a98b00df4929841903fc 2014-12-23T17:42:42.237+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a98b00df4929841903fc, took 0ms 2014-12-23T17:42:42.237+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a98b00df4929841903fc 2014-12-23T17:42:42.486+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:42-5499a9929a8f659f34bfb949", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356562486), what: "split", ns: "benchdb1.COL-21", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a98b00df4929841903fc') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a98b00df4929841903fc') } } } 2014-12-23T17:42:42.706+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-21/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:42.706+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-21", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-21-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a98b00df4929841903fc') } ntoreturn:1 keyUpdates:0 reslen:37 938ms 2014-12-23T17:42:44.888+0000 I INDEX [conn2] build index on: benchdb1.COL-22 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-22" } 2014-12-23T17:42:44.888+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:42:44.889+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:45.224+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-22 with requested shard version 1|2||5499a99100df492984190401, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:45.225+0000 I SHARDING [conn9] collection benchdb1.COL-22 was previously unsharded, new metadata loaded with shard version 1|2||5499a99100df492984190401 2014-12-23T17:42:45.225+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a99100df492984190401, took 0ms 2014-12-23T17:42:45.274+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:45.274+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99100df492984190401') } 2014-12-23T17:42:45.397+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-6 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:42:45.616+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-6 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:42:45.881+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9959a8f659f34bfb94b 2014-12-23T17:42:45.882+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:45-5499a9959a8f659f34bfb94c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356565882), what: "moveChunk.start", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:45.965+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 1|2||5499a99100df492984190401, current metadata version is 1|2||5499a99100df492984190401 2014-12-23T17:42:45.966+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-22 already up to date (shard version : 1|2||5499a99100df492984190401, took 0ms) 2014-12-23T17:42:45.966+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a99100df492984190401 2014-12-23T17:42:45.967+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:45.970+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:45.972+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:45.976+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:45.985+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:46.001+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:46.001+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:46.002+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:46.002+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a99100df492984190401 2014-12-23T17:42:46.073+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:46.074+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a99100df492984190401 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-22' 2014-12-23T17:42:46.156+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-6 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:42:46.379+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:46-5499a9969a8f659f34bfb94d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356566379), what: "moveChunk.commit", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:46.447+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:46.455+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:46.455+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-22 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 21124486 2014-12-23T17:42:46.455+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-22 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:46.455+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:46.493+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-6 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:42:46.758+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:46.758+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:46-5499a9969a8f659f34bfb94e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356566758), what: "moveChunk.from", ns: "benchdb1.COL-22", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 691, step 3 of 6: 2, step 4 of 6: 33, step 5 of 6: 453, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:46.879+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99100df492984190401') } ntoreturn:1 keyUpdates:0 reslen:37 1604ms 2014-12-23T17:42:46.884+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:46.884+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99100df492984190401') } 2014-12-23T17:42:46.943+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-6 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:42:47.321+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-6 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:42:47.334+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4641102259449717728 }, { shardkey: -3125623704900401122 } ], shardId: "benchdb1.COL-6-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92100df4929841903b9') } 2014-12-23T17:42:47.443+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9969a8f659f34bfb94f 2014-12-23T17:42:47.443+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:47-5499a9979a8f659f34bfb951", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356567443), what: "moveChunk.start", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:47.519+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 2|0||5499a99100df492984190401, current metadata version is 2|0||5499a99100df492984190401 2014-12-23T17:42:47.520+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-22 from shard version 2|0||5499a99100df492984190401 to shard version 2|1||5499a99100df492984190401 2014-12-23T17:42:47.520+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a99100df492984190401, took 0ms 2014-12-23T17:42:47.520+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a99100df492984190401 2014-12-23T17:42:47.520+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:47.522+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:47.524+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:47.529+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:47.537+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:47.554+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:47.554+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:47.554+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:47.554+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a99100df492984190401 2014-12-23T17:42:47.612+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:47.612+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a99100df492984190401 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-22' 2014-12-23T17:42:47.764+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9979a8f659f34bfb950 2014-12-23T17:42:47.764+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 based on current shard version 3|3||5499a92100df4929841903b9, current metadata version is 3|3||5499a92100df4929841903b9 2014-12-23T17:42:47.765+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-6 from shard version 3|3||5499a92100df4929841903b9 to shard version 3|3||5499a92100df4929841903b9 2014-12-23T17:42:47.765+0000 I SHARDING [conn1] collection version was loaded at version 3|7||5499a92100df4929841903b9, took 1ms 2014-12-23T17:42:47.765+0000 I SHARDING [conn1] splitChunk accepted at version 3|3||5499a92100df4929841903b9 2014-12-23T17:42:47.917+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:47-5499a9979a8f659f34bfb952", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356567917), what: "moveChunk.commit", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:47.991+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:47.991+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:47.991+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-22 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 21172869 2014-12-23T17:42:47.991+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-22 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:47.991+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:48.042+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:48-5499a9989a8f659f34bfb953", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356568042), what: "multi-split", ns: "benchdb1.COL-6", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 1, of: 3, chunk: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -4641102259449717728 }, lastmod: Timestamp 3000|8, lastmodEpoch: ObjectId('5499a92100df4929841903b9') } } } 2014-12-23T17:42:48.295+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:48.295+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:48-5499a9989a8f659f34bfb954", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356568295), what: "moveChunk.from", ns: "benchdb1.COL-22", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 635, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 437, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:48.315+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:48-5499a9989a8f659f34bfb955", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356568315), what: "multi-split", ns: "benchdb1.COL-6", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 2, of: 3, chunk: { min: { shardkey: -4641102259449717728 }, max: { shardkey: -3125623704900401122 }, lastmod: Timestamp 3000|9, lastmodEpoch: ObjectId('5499a92100df4929841903b9') } } } 2014-12-23T17:42:48.356+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-22", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-22-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99100df492984190401') } ntoreturn:1 keyUpdates:0 reslen:37 1471ms 2014-12-23T17:42:48.360+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-22", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-22-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a99100df492984190401') } 2014-12-23T17:42:48.388+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:48-5499a9989a8f659f34bfb957", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356568388), what: "multi-split", ns: "benchdb1.COL-6", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 3, of: 3, chunk: { min: { shardkey: -3125623704900401122 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|10, lastmodEpoch: ObjectId('5499a92100df4929841903b9') } } } 2014-12-23T17:42:48.736+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-6/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:48.737+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-6", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4641102259449717728 }, { shardkey: -3125623704900401122 } ], shardId: "benchdb1.COL-6-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92100df4929841903b9') } ntoreturn:1 keyUpdates:0 reslen:37 1402ms 2014-12-23T17:42:48.893+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9989a8f659f34bfb956 2014-12-23T17:42:48.893+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 based on current shard version 3|0||5499a99100df492984190401, current metadata version is 3|0||5499a99100df492984190401 2014-12-23T17:42:48.894+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-22 from shard version 3|0||5499a99100df492984190401 to shard version 3|1||5499a99100df492984190401 2014-12-23T17:42:48.894+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a99100df492984190401, took 0ms 2014-12-23T17:42:48.894+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a99100df492984190401 2014-12-23T17:42:49.201+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:49-5499a9999a8f659f34bfb958", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356569201), what: "split", ns: "benchdb1.COL-22", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a99100df492984190401') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a99100df492984190401') } } } 2014-12-23T17:42:50.561+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-22/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:50.561+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-22", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-22-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a99100df492984190401') } ntoreturn:1 keyUpdates:0 reslen:37 2200ms 2014-12-23T17:42:50.765+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-6 { : -6148914691236517204 } -->> { : -4641102259449717728 } 2014-12-23T17:42:50.781+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-6 { : -4641102259449717728 } -->> { : -3125623704900401122 } 2014-12-23T17:42:50.796+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-6 { : -3125623704900401122 } -->> { : -3074457345618258602 } 2014-12-23T17:42:51.499+0000 I WRITE [conn2] insert benchdb1.COL-60 query: { _id: ObjectId('5498b73a0315362bd8727543'), shardkey: "R3N6fP1hgoE6ri1nYvu3ABSh", payload: "OPMKLtz8CL0GpwXFogO64F6QBLRbXWoq2prJMOUd1wr7b6bTMMnHXL91xNYVESkHdV7VXnsXv1uwbEfseTrFqJtrARJVgZMC8NkC6tWj67BEygWy54FLkadFKyADZ4lRy83ITMxYMHhgkAEIBS9P7z..." } ninserted:1 keyUpdates:0 530ms 2014-12-23T17:42:51.530+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-60", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 560ms 2014-12-23T17:42:52.426+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:42:51 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:42:53.633+0000 I INDEX [conn9] build index on: benchdb1.COL-23 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-23" } 2014-12-23T17:42:53.633+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:42:53.634+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:42:53.988+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-23 with requested shard version 1|2||5499a99a00df492984190405, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:42:53.989+0000 I SHARDING [conn9] collection benchdb1.COL-23 was previously unsharded, new metadata loaded with shard version 1|2||5499a99a00df492984190405 2014-12-23T17:42:53.989+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a99a00df492984190405, took 0ms 2014-12-23T17:42:54.036+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:54.036+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99a00df492984190405') } 2014-12-23T17:42:54.384+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a99e9a8f659f34bfb95a 2014-12-23T17:42:54.385+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:54-5499a99e9a8f659f34bfb95b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356574385), what: "moveChunk.start", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:42:54.419+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 1|2||5499a99a00df492984190405, current metadata version is 1|2||5499a99a00df492984190405 2014-12-23T17:42:54.420+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-23 already up to date (shard version : 1|2||5499a99a00df492984190405, took 0ms) 2014-12-23T17:42:54.420+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a99a00df492984190405 2014-12-23T17:42:54.430+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:54.433+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:54.442+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:54.442+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:54.442+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:54.442+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a99a00df492984190405 2014-12-23T17:42:54.512+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:54.512+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a99a00df492984190405 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-23' 2014-12-23T17:42:54.715+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:54-5499a99e9a8f659f34bfb95c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356574715), what: "moveChunk.commit", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:54.766+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:54.766+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:54.766+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-23 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 21315366 2014-12-23T17:42:54.766+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-23 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:42:54.767+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:54.951+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:54.952+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:54-5499a99e9a8f659f34bfb95d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356574952), what: "moveChunk.from", ns: "benchdb1.COL-23", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 384, step 3 of 6: 11, step 4 of 6: 9, step 5 of 6: 324, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:42:55.006+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99a00df492984190405') } ntoreturn:1 keyUpdates:0 reslen:37 969ms 2014-12-23T17:42:55.007+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:42:55.008+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99a00df492984190405') } 2014-12-23T17:42:55.510+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a99f9a8f659f34bfb95e 2014-12-23T17:42:55.510+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:55-5499a99f9a8f659f34bfb95f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356575510), what: "moveChunk.start", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:42:55.582+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 2|0||5499a99a00df492984190405, current metadata version is 2|0||5499a99a00df492984190405 2014-12-23T17:42:55.583+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-23 from shard version 2|0||5499a99a00df492984190405 to shard version 2|1||5499a99a00df492984190405 2014-12-23T17:42:55.583+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a99a00df492984190405, took 0ms 2014-12-23T17:42:55.583+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a99a00df492984190405 2014-12-23T17:42:55.605+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:42:55.608+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:55.610+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:55.615+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:55.623+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:55.640+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:42:55.640+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:42:55.640+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:42:55.640+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a99a00df492984190405 2014-12-23T17:42:55.716+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:42:55.717+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a99a00df492984190405 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-23' 2014-12-23T17:42:55.936+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a99a00df492984190405') } 2014-12-23T17:42:55.936+0000 W SHARDING [conn1] migration already in progress 2014-12-23T17:42:55.936+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:55-5499a99f9a8f659f34bfb960", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356575936), what: "moveChunk.from", ns: "benchdb1.COL-23", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:42:55.979+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:55-5499a99f9a8f659f34bfb961", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356575979), what: "moveChunk.commit", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:42:56.027+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:56.034+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:42:56.034+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-23 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 21380605 2014-12-23T17:42:56.034+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-23 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:42:56.035+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:42:56.266+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:56.266+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:56-5499a9a09a8f659f34bfb962", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356576266), what: "moveChunk.from", ns: "benchdb1.COL-23", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 575, step 3 of 6: 22, step 4 of 6: 33, step 5 of 6: 394, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:42:56.322+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-23", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-23-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a99a00df492984190405') } ntoreturn:1 keyUpdates:0 reslen:37 1315ms 2014-12-23T17:42:56.324+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-23", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-23-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a99a00df492984190405') } 2014-12-23T17:42:56.931+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9a09a8f659f34bfb963 2014-12-23T17:42:56.931+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-23 based on current shard version 3|0||5499a99a00df492984190405, current metadata version is 3|0||5499a99a00df492984190405 2014-12-23T17:42:56.932+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-23 from shard version 3|0||5499a99a00df492984190405 to shard version 3|1||5499a99a00df492984190405 2014-12-23T17:42:56.932+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a99a00df492984190405, took 0ms 2014-12-23T17:42:56.932+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a99a00df492984190405 2014-12-23T17:42:57.237+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:42:57-5499a9a19a8f659f34bfb964", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356577237), what: "split", ns: "benchdb1.COL-23", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a99a00df492984190405') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a99a00df492984190405') } } } 2014-12-23T17:42:57.641+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-23/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:42:57.641+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-23", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-23-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a99a00df492984190405') } ntoreturn:1 keyUpdates:0 reslen:37 1317ms 2014-12-23T17:42:58.225+0000 I WRITE [conn2] insert benchdb1.COL-61 query: { _id: ObjectId('5498b7500315362bd8758174'), shardkey: "MKnCZVYm8JQ2dJ7FVUrziTaB", payload: "o5pnpQHoMpABEw6BExucLhPmz0NOKm5CLGViCF2bc8WKiuuFCsYxyLLsM70OFJxSZjb5ofM2SwBgOi6SFoYoI3Jx3fJvGvPsxhANGOIVzscNI57bpswngwL4aFG3si4UxNlhcl9AaoDTGfxa1TstDH..." } ninserted:1 keyUpdates:0 286ms 2014-12-23T17:42:58.241+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-61", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 302ms 2014-12-23T17:43:00.194+0000 I INDEX [conn2] build index on: benchdb1.COL-24 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-24" } 2014-12-23T17:43:00.194+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:43:00.195+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:00.517+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-24 with requested shard version 1|2||5499a9a100df492984190409, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:00.518+0000 I SHARDING [conn9] collection benchdb1.COL-24 was previously unsharded, new metadata loaded with shard version 1|2||5499a9a100df492984190409 2014-12-23T17:43:00.518+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a9a100df492984190409, took 0ms 2014-12-23T17:43:00.560+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:00.561+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a100df492984190409') } 2014-12-23T17:43:01.168+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9a49a8f659f34bfb966 2014-12-23T17:43:01.168+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:01-5499a9a59a8f659f34bfb967", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356581168), what: "moveChunk.start", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:01.229+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 1|2||5499a9a100df492984190409, current metadata version is 1|2||5499a9a100df492984190409 2014-12-23T17:43:01.230+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-24 already up to date (shard version : 1|2||5499a9a100df492984190409, took 0ms) 2014-12-23T17:43:01.230+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a9a100df492984190409 2014-12-23T17:43:01.231+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:01.233+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:01.236+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:01.240+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:01.240+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:01.241+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:01.241+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a9a100df492984190409 2014-12-23T17:43:01.296+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:01.297+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a9a100df492984190409 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-24' 2014-12-23T17:43:01.602+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:01-5499a9a59a8f659f34bfb968", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356581602), what: "moveChunk.commit", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:01.673+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:01.679+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:01.679+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-24 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 21730790 2014-12-23T17:43:01.679+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-24 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:43:01.679+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:01.982+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:01.982+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:01-5499a9a59a8f659f34bfb969", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356581982), what: "moveChunk.from", ns: "benchdb1.COL-24", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 669, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 438, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:43:02.049+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a100df492984190409') } ntoreturn:1 keyUpdates:0 reslen:37 1488ms 2014-12-23T17:43:02.050+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:02.051+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a100df492984190409') } 2014-12-23T17:43:02.658+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9a69a8f659f34bfb96a 2014-12-23T17:43:02.658+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:02-5499a9a69a8f659f34bfb96b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356582658), what: "moveChunk.start", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:02.727+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 2|0||5499a9a100df492984190409, current metadata version is 2|0||5499a9a100df492984190409 2014-12-23T17:43:02.728+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-24 from shard version 2|0||5499a9a100df492984190409 to shard version 2|1||5499a9a100df492984190409 2014-12-23T17:43:02.728+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a9a100df492984190409, took 0ms 2014-12-23T17:43:02.728+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a9a100df492984190409 2014-12-23T17:43:02.728+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:02.731+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:02.733+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:02.737+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:02.746+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:02.762+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:02.762+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:02.763+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:02.763+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a9a100df492984190409 2014-12-23T17:43:02.849+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:02.849+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a9a100df492984190409 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-24' 2014-12-23T17:43:03.155+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:03-5499a9a79a8f659f34bfb96c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356583155), what: "moveChunk.commit", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:03.431+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:03.432+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:03.432+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-24 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 21829681 2014-12-23T17:43:03.432+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-24 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:43:03.432+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:03.736+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:03.736+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:03-5499a9a79a8f659f34bfb96d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356583736), what: "moveChunk.from", ns: "benchdb1.COL-24", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 677, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 669, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:03.795+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-24", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-24-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a100df492984190409') } ntoreturn:1 keyUpdates:0 reslen:37 1744ms 2014-12-23T17:43:03.797+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-24", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-24-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9a100df492984190409') } 2014-12-23T17:43:04.405+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9a79a8f659f34bfb96e 2014-12-23T17:43:04.405+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-24 based on current shard version 3|0||5499a9a100df492984190409, current metadata version is 3|0||5499a9a100df492984190409 2014-12-23T17:43:04.406+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-24 from shard version 3|0||5499a9a100df492984190409 to shard version 3|1||5499a9a100df492984190409 2014-12-23T17:43:04.406+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a9a100df492984190409, took 0ms 2014-12-23T17:43:04.406+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a9a100df492984190409 2014-12-23T17:43:04.777+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:04-5499a9a89a8f659f34bfb96f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356584777), what: "split", ns: "benchdb1.COL-24", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a9a100df492984190409') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a9a100df492984190409') } } } 2014-12-23T17:43:05.159+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-24/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:05.160+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-24", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-24-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9a100df492984190409') } ntoreturn:1 keyUpdates:0 reslen:37 1362ms 2014-12-23T17:43:07.412+0000 I INDEX [conn9] build index on: benchdb1.COL-25 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-25" } 2014-12-23T17:43:07.412+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:43:07.413+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:07.731+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-25 with requested shard version 1|2||5499a9a800df49298419040d, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:07.732+0000 I SHARDING [conn9] collection benchdb1.COL-25 was previously unsharded, new metadata loaded with shard version 1|2||5499a9a800df49298419040d 2014-12-23T17:43:07.732+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a9a800df49298419040d, took 0ms 2014-12-23T17:43:07.778+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:07.779+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:08.116+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9ab9a8f659f34bfb971 2014-12-23T17:43:08.116+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:08-5499a9ac9a8f659f34bfb972", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356588116), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:08.179+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 1|2||5499a9a800df49298419040d, current metadata version is 1|2||5499a9a800df49298419040d 2014-12-23T17:43:08.180+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-25 already up to date (shard version : 1|2||5499a9a800df49298419040d, took 0ms) 2014-12-23T17:43:08.180+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a9a800df49298419040d 2014-12-23T17:43:08.193+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:08.201+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:08.233+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:08.233+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:08.233+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:08.233+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a9a800df49298419040d 2014-12-23T17:43:08.316+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:08.316+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a9a800df49298419040d through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-25' 2014-12-23T17:43:08.616+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:08-5499a9ac9a8f659f34bfb973", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356588616), what: "moveChunk.commit", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:08.694+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:08.694+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:08.694+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-25 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 22194556 2014-12-23T17:43:08.694+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-25 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:43:08.694+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:08.948+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:08.948+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:08-5499a9ac9a8f659f34bfb974", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356588948), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 401, step 3 of 6: 14, step 4 of 6: 38, step 5 of 6: 460, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:43:08.998+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } ntoreturn:1 keyUpdates:0 reslen:37 1219ms 2014-12-23T17:43:08.999+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:09.000+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:09.609+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9ac9a8f659f34bfb975 2014-12-23T17:43:09.609+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:09-5499a9ad9a8f659f34bfb976", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356589609), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:09.731+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 2|0||5499a9a800df49298419040d, current metadata version is 2|0||5499a9a800df49298419040d 2014-12-23T17:43:09.732+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-25 from shard version 2|0||5499a9a800df49298419040d to shard version 2|1||5499a9a800df49298419040d 2014-12-23T17:43:09.732+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a9a800df49298419040d, took 1ms 2014-12-23T17:43:09.732+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a9a800df49298419040d 2014-12-23T17:43:09.747+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:09.747+0000 W SHARDING [conn4] moveChunk failed to engage TO-shard in the data transfer: migrate already in progress 2014-12-23T17:43:09.747+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:10.051+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:10.051+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:10-5499a9ae9a8f659f34bfb977", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356590051), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 731, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "moveChunk failed to engage TO-shard in the data transfer: migrate already in progress" } } 2014-12-23T17:43:10.119+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } ntoreturn:1 keyUpdates:0 reslen:199 1119ms 2014-12-23T17:43:10.120+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:10.727+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9ae9a8f659f34bfb978 2014-12-23T17:43:10.727+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 2|1||5499a9a800df49298419040d, current metadata version is 2|1||5499a9a800df49298419040d 2014-12-23T17:43:10.728+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-25 already up to date (shard version : 2|1||5499a9a800df49298419040d, took 0ms) 2014-12-23T17:43:10.728+0000 I SHARDING [conn4] splitChunk accepted at version 2|1||5499a9a800df49298419040d 2014-12-23T17:43:11.033+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:11-5499a9af9a8f659f34bfb979", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356591033), what: "split", ns: "benchdb1.COL-25", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('5499a9a800df49298419040d') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('5499a9a800df49298419040d') } } } 2014-12-23T17:43:11.304+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:11.304+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9a800df49298419040d') } ntoreturn:1 keyUpdates:0 reslen:37 1183ms 2014-12-23T17:43:12.487+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:13.094+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b09a8f659f34bfb97a 2014-12-23T17:43:13.095+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 2|3||5499a9a800df49298419040d, current metadata version is 2|3||5499a9a800df49298419040d 2014-12-23T17:43:13.096+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-25 from shard version 2|3||5499a9a800df49298419040d to shard version 2|3||5499a9a800df49298419040d 2014-12-23T17:43:13.096+0000 I SHARDING [conn4] collection version was loaded at version 2|5||5499a9a800df49298419040d, took 1ms 2014-12-23T17:43:13.096+0000 I SHARDING [conn4] splitChunk accepted at version 2|3||5499a9a800df49298419040d 2014-12-23T17:43:13.318+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:13-5499a9b19a8f659f34bfb97b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356593318), what: "split", ns: "benchdb1.COL-25", details: { before: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey } }, left: { min: { shardkey: 3074457345618258602 }, max: { shardkey: 6148914691236517204 }, lastmod: Timestamp 2000|6, lastmodEpoch: ObjectId('5499a9a800df49298419040d') }, right: { min: { shardkey: 6148914691236517204 }, max: { shardkey: MaxKey }, lastmod: Timestamp 2000|7, lastmodEpoch: ObjectId('5499a9a800df49298419040d') } } } 2014-12-23T17:43:13.934+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:13.934+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-25", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-25-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9a800df49298419040d') } ntoreturn:1 keyUpdates:0 reslen:37 1447ms 2014-12-23T17:43:13.953+0000 I INDEX [conn2] build index on: benchdb1.COL-26 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-26" } 2014-12-23T17:43:13.953+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:43:13.954+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:14.239+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-26 with requested shard version 1|2||5499a9ae00df492984190412, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:14.240+0000 I SHARDING [conn9] collection benchdb1.COL-26 was previously unsharded, new metadata loaded with shard version 1|2||5499a9ae00df492984190412 2014-12-23T17:43:14.240+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a9ae00df492984190412, took 0ms 2014-12-23T17:43:14.284+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:14.284+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:14.565+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b29a8f659f34bfb97d 2014-12-23T17:43:14.565+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:14-5499a9b29a8f659f34bfb97e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356594565), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:14.619+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 1|2||5499a9ae00df492984190412, current metadata version is 1|2||5499a9ae00df492984190412 2014-12-23T17:43:14.620+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-26 already up to date (shard version : 1|2||5499a9ae00df492984190412, took 0ms) 2014-12-23T17:43:14.620+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a9ae00df492984190412 2014-12-23T17:43:14.632+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:14.635+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:14.637+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:14.642+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:14.650+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:14.650+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:14.651+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:14.651+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a9ae00df492984190412 2014-12-23T17:43:14.703+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:14.704+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a9ae00df492984190412 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-26' 2014-12-23T17:43:14.948+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:14-5499a9b29a8f659f34bfb97f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356594948), what: "moveChunk.commit", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:15.004+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:15.011+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:15.011+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-26 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 22638149 2014-12-23T17:43:15.011+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-26 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:43:15.011+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:15.219+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:15.219+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:15-5499a9b39a8f659f34bfb980", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356595219), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 335, step 3 of 6: 13, step 4 of 6: 16, step 5 of 6: 360, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:43:15.221+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:15.316+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } ntoreturn:1 keyUpdates:0 reslen:37 1031ms 2014-12-23T17:43:15.317+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:15.318+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:15.775+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b39a8f659f34bfb981 2014-12-23T17:43:15.775+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:15-5499a9b39a8f659f34bfb983", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356595775), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:15.845+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 2|7||5499a9a800df49298419040d, current metadata version is 2|7||5499a9a800df49298419040d 2014-12-23T17:43:15.845+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b39a8f659f34bfb982 2014-12-23T17:43:15.845+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:15-5499a9b39a8f659f34bfb984", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356595845), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:15.846+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-25 already up to date (shard version : 2|7||5499a9a800df49298419040d, took 1ms) 2014-12-23T17:43:15.846+0000 I SHARDING [conn1] moveChunk request accepted at version 2|7||5499a9a800df49298419040d 2014-12-23T17:43:15.846+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:43:15.848+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:15.851+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:15.855+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:15.864+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:15.881+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:15.881+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:43:15.881+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:43:15.881+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a9a800df49298419040d 2014-12-23T17:43:15.926+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 2|0||5499a9ae00df492984190412, current metadata version is 2|0||5499a9ae00df492984190412 2014-12-23T17:43:15.927+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-26 from shard version 2|0||5499a9ae00df492984190412 to shard version 2|1||5499a9ae00df492984190412 2014-12-23T17:43:15.927+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a9ae00df492984190412, took 0ms 2014-12-23T17:43:15.927+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a9ae00df492984190412 2014-12-23T17:43:15.927+0000 W SHARDING [conn4] moveChunk is already in progress from this shard 2014-12-23T17:43:15.980+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:15.980+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a9a800df49298419040d through { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-25' 2014-12-23T17:43:16.135+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:16.135+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:16-5499a9b49a8f659f34bfb985", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356596135), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 608, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "moveChunk is already in progress from this shard" } } 2014-12-23T17:43:16.201+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } ntoreturn:1 keyUpdates:0 reslen:98 883ms 2014-12-23T17:43:16.202+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:16.251+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:16-5499a9b49a8f659f34bfb987", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356596251), what: "moveChunk.commit", ns: "benchdb1.COL-25", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:16.301+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:16.302+0000 I SHARDING [conn1] forking for cleanup of chunk data 2014-12-23T17:43:16.302+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:16.302+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-25 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 }, with opId: 3 2014-12-23T17:43:16.302+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-25 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 } 2014-12-23T17:43:16.450+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:42214 #10 (10 connections now open) 2014-12-23T17:43:16.605+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:16.605+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:16-5499a9b49a8f659f34bfb988", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356596605), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, step 2 of 6: 625, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 421, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:16.673+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } ntoreturn:1 keyUpdates:0 reslen:37 1452ms 2014-12-23T17:43:16.738+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b49a8f659f34bfb986 2014-12-23T17:43:16.738+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 2|1||5499a9ae00df492984190412, current metadata version is 2|1||5499a9ae00df492984190412 2014-12-23T17:43:16.739+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-26 already up to date (shard version : 2|1||5499a9ae00df492984190412, took 0ms) 2014-12-23T17:43:16.740+0000 I SHARDING [conn4] splitChunk accepted at version 2|1||5499a9ae00df492984190412 2014-12-23T17:43:17.046+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:17-5499a9b59a8f659f34bfb989", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356597046), what: "split", ns: "benchdb1.COL-26", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('5499a9ae00df492984190412') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('5499a9ae00df492984190412') } } } 2014-12-23T17:43:17.443+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:17.443+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9ae00df492984190412') } ntoreturn:1 keyUpdates:0 reslen:37 1240ms 2014-12-23T17:43:18.890+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:19.080+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:43:19.198+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b69a8f659f34bfb98a 2014-12-23T17:43:19.198+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 2|3||5499a9ae00df492984190412, current metadata version is 2|3||5499a9ae00df492984190412 2014-12-23T17:43:19.199+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-26 from shard version 2|3||5499a9ae00df492984190412 to shard version 2|3||5499a9ae00df492984190412 2014-12-23T17:43:19.199+0000 I SHARDING [conn4] collection version was loaded at version 2|5||5499a9ae00df492984190412, took 0ms 2014-12-23T17:43:19.199+0000 I SHARDING [conn4] splitChunk accepted at version 2|3||5499a9ae00df492984190412 2014-12-23T17:43:19.341+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:43:19.379+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:19-5499a9b79a8f659f34bfb98b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356599379), what: "split", ns: "benchdb1.COL-26", details: { before: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey } }, left: { min: { shardkey: 3074457345618258602 }, max: { shardkey: 6148914691236517204 }, lastmod: Timestamp 2000|6, lastmodEpoch: ObjectId('5499a9ae00df492984190412') }, right: { min: { shardkey: 6148914691236517204 }, max: { shardkey: MaxKey }, lastmod: Timestamp 2000|7, lastmodEpoch: ObjectId('5499a9ae00df492984190412') } } } 2014-12-23T17:43:19.596+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:19.596+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-26", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-26-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9ae00df492984190412') } ntoreturn:1 keyUpdates:0 reslen:37 706ms 2014-12-23T17:43:19.607+0000 I INDEX [conn2] build index on: benchdb1.COL-27 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-27" } 2014-12-23T17:43:19.608+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:43:19.608+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:19.922+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-27 with requested shard version 1|2||5499a9b400df492984190417, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:19.923+0000 I SHARDING [conn9] collection benchdb1.COL-27 was previously unsharded, new metadata loaded with shard version 1|2||5499a9b400df492984190417 2014-12-23T17:43:19.923+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a9b400df492984190417, took 0ms 2014-12-23T17:43:19.968+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:19.968+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9b400df492984190417') } 2014-12-23T17:43:20.237+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b79a8f659f34bfb98d 2014-12-23T17:43:20.237+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:20-5499a9b89a8f659f34bfb98e", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356600237), what: "moveChunk.start", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:20.297+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 1|2||5499a9b400df492984190417, current metadata version is 1|2||5499a9b400df492984190417 2014-12-23T17:43:20.298+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-27 already up to date (shard version : 1|2||5499a9b400df492984190417, took 0ms) 2014-12-23T17:43:20.298+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a9b400df492984190417 2014-12-23T17:43:20.298+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:20.301+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:20.303+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:20.308+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:20.308+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:20.308+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:20.308+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a9b400df492984190417 2014-12-23T17:43:20.367+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:20.367+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a9b400df492984190417 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-27' 2014-12-23T17:43:20.380+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:43:20.414+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:20.415+0000 W SHARDING [conn1] migration already in progress 2014-12-23T17:43:20.415+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:20-5499a9b89a8f659f34bfb98f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356600415), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:43:20.454+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:20.454+0000 W SHARDING [conn1] migration already in progress 2014-12-23T17:43:20.454+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:20-5499a9b89a8f659f34bfb990", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356600454), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "migration already in progress" } } 2014-12-23T17:43:20.580+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:20-5499a9b89a8f659f34bfb991", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356600580), what: "moveChunk.commit", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:20.631+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:20.631+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:20.632+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-27 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 22876377 2014-12-23T17:43:20.632+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-27 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:43:20.632+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:20.764+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:43:20.839+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:20.839+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:20-5499a9b89a8f659f34bfb992", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356600839), what: "moveChunk.from", ns: "benchdb1.COL-27", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 329, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 323, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:43:20.916+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9b400df492984190417') } ntoreturn:1 keyUpdates:0 reslen:37 948ms 2014-12-23T17:43:20.918+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:20.918+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9b400df492984190417') } 2014-12-23T17:43:21.304+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:43:21.525+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b89a8f659f34bfb993 2014-12-23T17:43:21.526+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:21-5499a9b99a8f659f34bfb994", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356601526), what: "moveChunk.start", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:21.610+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 2|0||5499a9b400df492984190417, current metadata version is 2|0||5499a9b400df492984190417 2014-12-23T17:43:21.610+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-27 from shard version 2|0||5499a9b400df492984190417 to shard version 2|1||5499a9b400df492984190417 2014-12-23T17:43:21.610+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a9b400df492984190417, took 0ms 2014-12-23T17:43:21.610+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a9b400df492984190417 2014-12-23T17:43:21.611+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:21.613+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:21.615+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:21.620+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:21.628+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:21.645+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:21.645+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:21.645+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:21.645+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a9b400df492984190417 2014-12-23T17:43:21.706+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:21.706+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a9b400df492984190417 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-27' 2014-12-23T17:43:21.779+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:43:21.793+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4664957305148829219 }, { shardkey: -3144731377115850958 } ], shardId: "benchdb1.COL-7-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92900df4929841903be') } 2014-12-23T17:43:22.013+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:22-5499a9ba9a8f659f34bfb996", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356602013), what: "moveChunk.commit", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:22.084+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:22.084+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:22.084+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-27 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 22897180 2014-12-23T17:43:22.084+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-27 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:43:22.084+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:22.287+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9b99a8f659f34bfb995 2014-12-23T17:43:22.287+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 based on current shard version 3|3||5499a92900df4929841903be, current metadata version is 3|3||5499a92900df4929841903be 2014-12-23T17:43:22.288+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-7 from shard version 3|3||5499a92900df4929841903be to shard version 3|3||5499a92900df4929841903be 2014-12-23T17:43:22.288+0000 I SHARDING [conn1] collection version was loaded at version 3|7||5499a92900df4929841903be, took 0ms 2014-12-23T17:43:22.288+0000 I SHARDING [conn1] splitChunk accepted at version 3|3||5499a92900df4929841903be 2014-12-23T17:43:22.387+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:22.387+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:22-5499a9ba9a8f659f34bfb997", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356602387), what: "moveChunk.from", ns: "benchdb1.COL-27", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 692, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 439, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:22.453+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-27", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-27-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9b400df492984190417') } ntoreturn:1 keyUpdates:0 reslen:37 1534ms 2014-12-23T17:43:22.457+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-27", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-27-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9b400df492984190417') } 2014-12-23T17:43:22.556+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:22-5499a9ba9a8f659f34bfb999", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356602556), what: "multi-split", ns: "benchdb1.COL-7", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 1, of: 3, chunk: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -4664957305148829219 }, lastmod: Timestamp 3000|8, lastmodEpoch: ObjectId('5499a92900df4929841903be') } } } 2014-12-23T17:43:22.631+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:22-5499a9ba9a8f659f34bfb99a", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356602631), what: "multi-split", ns: "benchdb1.COL-7", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 2, of: 3, chunk: { min: { shardkey: -4664957305148829219 }, max: { shardkey: -3144731377115850958 }, lastmod: Timestamp 3000|9, lastmodEpoch: ObjectId('5499a92900df4929841903be') } } } 2014-12-23T17:43:22.696+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:43:22 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:43:22.696+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:22-5499a9ba9a8f659f34bfb99b", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356602696), what: "multi-split", ns: "benchdb1.COL-7", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 3, of: 3, chunk: { min: { shardkey: -3144731377115850958 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|10, lastmodEpoch: ObjectId('5499a92900df4929841903be') } } } 2014-12-23T17:43:22.947+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9ba9a8f659f34bfb998 2014-12-23T17:43:22.947+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-27 based on current shard version 3|0||5499a9b400df492984190417, current metadata version is 3|0||5499a9b400df492984190417 2014-12-23T17:43:22.948+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-27 from shard version 3|0||5499a9b400df492984190417 to shard version 3|1||5499a9b400df492984190417 2014-12-23T17:43:22.948+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a9b400df492984190417, took 0ms 2014-12-23T17:43:22.948+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a9b400df492984190417 2014-12-23T17:43:23.047+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-7/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:23.047+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-7", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4664957305148829219 }, { shardkey: -3144731377115850958 } ], shardId: "benchdb1.COL-7-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a92900df4929841903be') } ntoreturn:1 keyUpdates:0 reslen:37 1253ms 2014-12-23T17:43:23.064+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : -6148914691236517204 } -->> { : -4664957305148829219 } 2014-12-23T17:43:23.074+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : -4664957305148829219 } -->> { : -3144731377115850958 } 2014-12-23T17:43:23.083+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-7 { : -3144731377115850958 } -->> { : -3074457345618258602 } 2014-12-23T17:43:23.252+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:23-5499a9bb9a8f659f34bfb99c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356603252), what: "split", ns: "benchdb1.COL-27", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a9b400df492984190417') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a9b400df492984190417') } } } 2014-12-23T17:43:23.571+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-27/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:23.571+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-27", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-27-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9b400df492984190417') } ntoreturn:1 keyUpdates:0 reslen:37 1114ms 2014-12-23T17:43:24.388+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-7 { : -3144731377115850958 } -->> { : -3074457345618258602 } 2014-12-23T17:43:25.994+0000 I INDEX [conn9] build index on: benchdb1.COL-28 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-28" } 2014-12-23T17:43:25.994+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:43:25.995+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:26.294+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-28 with requested shard version 1|2||5499a9ba00df49298419041b, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:26.294+0000 I SHARDING [conn2] collection benchdb1.COL-28 was previously unsharded, new metadata loaded with shard version 1|2||5499a9ba00df49298419041b 2014-12-23T17:43:26.294+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a9ba00df49298419041b, took 0ms 2014-12-23T17:43:26.340+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:26.341+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ba00df49298419041b') } 2014-12-23T17:43:26.947+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9be9a8f659f34bfb99e 2014-12-23T17:43:26.948+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:26-5499a9be9a8f659f34bfb99f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356606948), what: "moveChunk.start", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:27.037+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 1|2||5499a9ba00df49298419041b, current metadata version is 1|2||5499a9ba00df49298419041b 2014-12-23T17:43:27.038+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-28 already up to date (shard version : 1|2||5499a9ba00df49298419041b, took 0ms) 2014-12-23T17:43:27.038+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a9ba00df49298419041b 2014-12-23T17:43:27.040+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:27.043+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:27.046+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:27.050+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:27.059+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:27.076+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:27.076+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:27.076+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:27.076+0000 I SHARDING [conn4] moveChunk setting version to: 2|0||5499a9ba00df49298419041b 2014-12-23T17:43:27.135+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:27.135+0000 I SHARDING [conn4] moveChunk updating self version to: 2|1||5499a9ba00df49298419041b through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-28' 2014-12-23T17:43:27.294+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:43:27.441+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:27-5499a9bf9a8f659f34bfb9a0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356607441), what: "moveChunk.commit", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:27.513+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:27.513+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:27.513+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-28 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 22968942 2014-12-23T17:43:27.513+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-28 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:43:27.513+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:27.538+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:43:27.817+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:27.817+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:27-5499a9bf9a8f659f34bfb9a1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356607817), what: "moveChunk.from", ns: "benchdb1.COL-28", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 697, step 3 of 6: 3, step 4 of 6: 33, step 5 of 6: 436, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:43:27.897+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ba00df49298419041b') } ntoreturn:1 keyUpdates:0 reslen:37 1556ms 2014-12-23T17:43:27.901+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:27.901+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ba00df49298419041b') } 2014-12-23T17:43:28.228+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:43:28.508+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9bf9a8f659f34bfb9a2 2014-12-23T17:43:28.508+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:28-5499a9c09a8f659f34bfb9a3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356608508), what: "moveChunk.start", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:28.560+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:43:28.584+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 2|0||5499a9ba00df49298419041b, current metadata version is 2|0||5499a9ba00df49298419041b 2014-12-23T17:43:28.585+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-28 from shard version 2|0||5499a9ba00df49298419041b to shard version 2|1||5499a9ba00df49298419041b 2014-12-23T17:43:28.585+0000 I SHARDING [conn4] collection version was loaded at version 2|1||5499a9ba00df49298419041b, took 0ms 2014-12-23T17:43:28.586+0000 I SHARDING [conn4] moveChunk request accepted at version 2|1||5499a9ba00df49298419041b 2014-12-23T17:43:28.606+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:28.609+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:28.611+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:28.616+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:28.624+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:28.640+0000 I SHARDING [conn4] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:28.640+0000 I SHARDING [conn4] About to check if it is safe to enter critical section 2014-12-23T17:43:28.641+0000 I SHARDING [conn4] About to enter migrate critical section 2014-12-23T17:43:28.641+0000 I SHARDING [conn4] moveChunk setting version to: 3|0||5499a9ba00df49298419041b 2014-12-23T17:43:28.710+0000 I SHARDING [conn4] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:28.710+0000 I SHARDING [conn4] moveChunk updating self version to: 3|1||5499a9ba00df49298419041b through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-28' 2014-12-23T17:43:29.017+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:29-5499a9c19a8f659f34bfb9a4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356609017), what: "moveChunk.commit", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:29.089+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:29.089+0000 I SHARDING [conn4] doing delete inline for cleanup of chunk data 2014-12-23T17:43:29.089+0000 I SHARDING [conn4] Deleter starting delete for: benchdb1.COL-28 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 23011118 2014-12-23T17:43:29.089+0000 I SHARDING [conn4] rangeDeleter deleted 0 documents for benchdb1.COL-28 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:43:29.089+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:29.154+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : MinKey } -->> { : -6148914691236517204 } 2014-12-23T17:43:29.392+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:29.392+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:29-5499a9c19a8f659f34bfb9a5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356609392), what: "moveChunk.from", ns: "benchdb1.COL-28", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 684, step 3 of 6: 21, step 4 of 6: 33, step 5 of 6: 448, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:29.471+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-28", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-28-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9ba00df49298419041b') } ntoreturn:1 keyUpdates:0 reslen:37 1569ms 2014-12-23T17:43:29.475+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-28", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-28-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9ba00df49298419041b') } 2014-12-23T17:43:29.607+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : -6148914691236517204 } -->> { : -3074457345618258602 } 2014-12-23T17:43:29.620+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4619962636474444350 }, { shardkey: -3114573996836400205 } ], shardId: "benchdb1.COL-8-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93000df4929841903c3') } 2014-12-23T17:43:29.984+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c19a8f659f34bfb9a6 2014-12-23T17:43:29.984+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-28 based on current shard version 3|0||5499a9ba00df49298419041b, current metadata version is 3|0||5499a9ba00df49298419041b 2014-12-23T17:43:29.985+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-28 from shard version 3|0||5499a9ba00df49298419041b to shard version 3|1||5499a9ba00df49298419041b 2014-12-23T17:43:29.985+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a9ba00df49298419041b, took 0ms 2014-12-23T17:43:29.985+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a9ba00df49298419041b 2014-12-23T17:43:30.085+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c19a8f659f34bfb9a7 2014-12-23T17:43:30.085+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 based on current shard version 3|3||5499a93000df4929841903c3, current metadata version is 3|3||5499a93000df4929841903c3 2014-12-23T17:43:30.086+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-8 from shard version 3|3||5499a93000df4929841903c3 to shard version 3|3||5499a93000df4929841903c3 2014-12-23T17:43:30.086+0000 I SHARDING [conn1] collection version was loaded at version 3|7||5499a93000df4929841903c3, took 0ms 2014-12-23T17:43:30.086+0000 I SHARDING [conn1] splitChunk accepted at version 3|3||5499a93000df4929841903c3 2014-12-23T17:43:30.113+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:30-5499a9c29a8f659f34bfb9a8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356610113), what: "split", ns: "benchdb1.COL-28", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a9ba00df49298419041b') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a9ba00df49298419041b') } } } 2014-12-23T17:43:30.228+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:30-5499a9c29a8f659f34bfb9a9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356610228), what: "multi-split", ns: "benchdb1.COL-8", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 1, of: 3, chunk: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -4619962636474444350 }, lastmod: Timestamp 3000|8, lastmodEpoch: ObjectId('5499a93000df4929841903c3') } } } 2014-12-23T17:43:30.280+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:30-5499a9c29a8f659f34bfb9aa", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356610280), what: "multi-split", ns: "benchdb1.COL-8", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 2, of: 3, chunk: { min: { shardkey: -4619962636474444350 }, max: { shardkey: -3114573996836400205 }, lastmod: Timestamp 3000|9, lastmodEpoch: ObjectId('5499a93000df4929841903c3') } } } 2014-12-23T17:43:30.322+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:30-5499a9c29a8f659f34bfb9ab", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356610322), what: "multi-split", ns: "benchdb1.COL-8", details: { before: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 } }, number: 3, of: 3, chunk: { min: { shardkey: -3114573996836400205 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|10, lastmodEpoch: ObjectId('5499a93000df4929841903c3') } } } 2014-12-23T17:43:30.429+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-28/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:30.429+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-28", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-28-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9ba00df49298419041b') } ntoreturn:1 keyUpdates:0 reslen:37 953ms 2014-12-23T17:43:30.632+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-8/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:30.633+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-8", keyPattern: { shardkey: "hashed" }, min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -4619962636474444350 }, { shardkey: -3114573996836400205 } ], shardId: "benchdb1.COL-8-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93000df4929841903c3') } ntoreturn:1 keyUpdates:0 reslen:37 1012ms 2014-12-23T17:43:30.649+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : -6148914691236517204 } -->> { : -4619962636474444350 } 2014-12-23T17:43:30.658+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : -4619962636474444350 } -->> { : -3114573996836400205 } 2014-12-23T17:43:30.667+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : -3114573996836400205 } -->> { : -3074457345618258602 } 2014-12-23T17:43:31.697+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-8 { : -3114573996836400205 } -->> { : -3074457345618258602 } 2014-12-23T17:43:32.173+0000 I INDEX [conn2] build index on: benchdb1.COL-29 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-29" } 2014-12-23T17:43:32.173+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:43:32.174+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:32.483+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-29 with requested shard version 1|2||5499a9c100df49298419041f, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:32.484+0000 I SHARDING [conn2] collection benchdb1.COL-29 was previously unsharded, new metadata loaded with shard version 1|2||5499a9c100df49298419041f 2014-12-23T17:43:32.484+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a9c100df49298419041f, took 1ms 2014-12-23T17:43:32.572+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:43:32.573+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c100df49298419041f') } 2014-12-23T17:43:32.890+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c49a8f659f34bfb9ad 2014-12-23T17:43:32.890+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:32-5499a9c49a8f659f34bfb9ae", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356612890), what: "moveChunk.start", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:32.937+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 1|2||5499a9c100df49298419041f, current metadata version is 1|2||5499a9c100df49298419041f 2014-12-23T17:43:32.938+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-29 already up to date (shard version : 1|2||5499a9c100df49298419041f, took 0ms) 2014-12-23T17:43:32.938+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a9c100df49298419041f 2014-12-23T17:43:32.938+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:43:32.941+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:32.943+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:32.948+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:32.948+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:43:32.948+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:43:32.948+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a9c100df49298419041f 2014-12-23T17:43:33.017+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:33.017+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a9c100df49298419041f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-29' 2014-12-23T17:43:33.165+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:33-5499a9c59a8f659f34bfb9af", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356613165), what: "moveChunk.commit", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:33.212+0000 I COMMAND [conn9] CMD: drop benchdb1.COL-4 2014-12-23T17:43:33.217+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:42482 #11 (11 connections now open) 2014-12-23T17:43:33.252+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:33.254+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:43:33.254+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-29 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 23068143 2014-12-23T17:43:33.254+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-29 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:43:33.254+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:33.556+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:33.556+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:33-5499a9c59a8f659f34bfb9b0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356613556), what: "moveChunk.from", ns: "benchdb1.COL-29", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 364, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 306, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:43:33.638+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c100df49298419041f') } ntoreturn:1 keyUpdates:0 reslen:37 1065ms 2014-12-23T17:43:33.641+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:43:33.642+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c100df49298419041f') } 2014-12-23T17:43:34.208+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c59a8f659f34bfb9b1 2014-12-23T17:43:34.208+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:34-5499a9c69a8f659f34bfb9b2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356614208), what: "moveChunk.start", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:34.277+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 2|0||5499a9c100df49298419041f, current metadata version is 2|0||5499a9c100df49298419041f 2014-12-23T17:43:34.278+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-29 from shard version 2|0||5499a9c100df49298419041f to shard version 2|1||5499a9c100df49298419041f 2014-12-23T17:43:34.278+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a9c100df49298419041f, took 0ms 2014-12-23T17:43:34.278+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a9c100df49298419041f 2014-12-23T17:43:34.278+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:43:34.280+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:34.283+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:34.287+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:34.295+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:34.312+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:34.312+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:43:34.312+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:43:34.312+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a9c100df49298419041f 2014-12-23T17:43:34.375+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:34.375+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a9c100df49298419041f through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-29' 2014-12-23T17:43:34.680+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:34-5499a9c69a8f659f34bfb9b3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356614680), what: "moveChunk.commit", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:34.747+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:34.749+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:43:34.749+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-29 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 23097646 2014-12-23T17:43:34.749+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-29 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:43:34.749+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:35.003+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:35.003+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:35-5499a9c79a8f659f34bfb9b4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356615003), what: "moveChunk.from", ns: "benchdb1.COL-29", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 635, step 3 of 6: 0, step 4 of 6: 33, step 5 of 6: 436, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:35.074+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-29", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-29-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c100df49298419041f') } ntoreturn:1 keyUpdates:0 reslen:37 1432ms 2014-12-23T17:43:35.078+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-29", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-29-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c100df49298419041f') } 2014-12-23T17:43:35.588+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c79a8f659f34bfb9b5 2014-12-23T17:43:35.588+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 based on current shard version 3|0||5499a9c100df49298419041f, current metadata version is 3|0||5499a9c100df49298419041f 2014-12-23T17:43:35.589+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-29 from shard version 3|0||5499a9c100df49298419041f to shard version 3|1||5499a9c100df49298419041f 2014-12-23T17:43:35.589+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a9c100df49298419041f, took 0ms 2014-12-23T17:43:35.589+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a9c100df49298419041f 2014-12-23T17:43:35.866+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:35-5499a9c79a8f659f34bfb9b6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356615866), what: "split", ns: "benchdb1.COL-29", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a9c100df49298419041f') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a9c100df49298419041f') } } } 2014-12-23T17:43:36.203+0000 I INDEX [conn11] build index on: benchdb1.COL-4 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-4" } 2014-12-23T17:43:36.203+0000 I INDEX [conn11] building index using bulk method 2014-12-23T17:43:36.204+0000 I INDEX [conn11] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:36.215+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-29/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:36.215+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-29", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-29-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c100df49298419041f') } ntoreturn:1 keyUpdates:0 reslen:37 1136ms 2014-12-23T17:43:36.503+0000 I SHARDING [conn11] remotely refreshing metadata for benchdb1.COL-4 with requested shard version 1|2||5499a9c500df492984190422, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:36.504+0000 I SHARDING [conn11] collection benchdb1.COL-4 was previously unsharded, new metadata loaded with shard version 1|2||5499a9c500df492984190422 2014-12-23T17:43:36.504+0000 I SHARDING [conn11] collection version was loaded at version 1|2||5499a9c500df492984190422, took 0ms 2014-12-23T17:43:36.550+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:43:36.550+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } 2014-12-23T17:43:36.631+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-9 { : 3074457345618258602 } -->> { : MaxKey } 2014-12-23T17:43:36.646+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6005881697089147565 }, { shardkey: 8919152878751157927 } ], shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93800df4929841903c6') } 2014-12-23T17:43:36.993+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c89a8f659f34bfb9b8 2014-12-23T17:43:36.993+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:36-5499a9c89a8f659f34bfb9ba", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356616993), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:37.003+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c89a8f659f34bfb9b9 2014-12-23T17:43:37.003+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-9 based on current shard version 3|0||5499a93800df4929841903c6, current metadata version is 3|0||5499a93800df4929841903c6 2014-12-23T17:43:37.004+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-9 from shard version 3|0||5499a93800df4929841903c6 to shard version 3|1||5499a93800df4929841903c6 2014-12-23T17:43:37.004+0000 I SHARDING [conn4] collection version was loaded at version 3|4||5499a93800df4929841903c6, took 0ms 2014-12-23T17:43:37.004+0000 I SHARDING [conn4] splitChunk accepted at version 3|1||5499a93800df4929841903c6 2014-12-23T17:43:37.114+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 1|2||5499a9c500df492984190422, current metadata version is 1|2||5499a9c500df492984190422 2014-12-23T17:43:37.116+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-4 already up to date (shard version : 1|2||5499a9c500df492984190422, took 1ms) 2014-12-23T17:43:37.116+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a9c500df492984190422 2014-12-23T17:43:37.116+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:43:37.119+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:37.122+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:37.126+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:37.134+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:37.134+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:43:37.135+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:43:37.135+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a9c500df492984190422 2014-12-23T17:43:37.207+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:37-5499a9c99a8f659f34bfb9bb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356617207), what: "multi-split", ns: "benchdb1.COL-9", details: { before: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey } }, number: 1, of: 3, chunk: { min: { shardkey: 3074457345618258602 }, max: { shardkey: 6005881697089147565 }, lastmod: Timestamp 3000|5, lastmodEpoch: ObjectId('5499a93800df4929841903c6') } } } 2014-12-23T17:43:37.267+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:37.267+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a9c500df492984190422 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-4' 2014-12-23T17:43:37.320+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:37-5499a9c99a8f659f34bfb9bc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356617320), what: "multi-split", ns: "benchdb1.COL-9", details: { before: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey } }, number: 2, of: 3, chunk: { min: { shardkey: 6005881697089147565 }, max: { shardkey: 8919152878751157927 }, lastmod: Timestamp 3000|6, lastmodEpoch: ObjectId('5499a93800df4929841903c6') } } } 2014-12-23T17:43:37.370+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:37-5499a9c99a8f659f34bfb9bd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356617370), what: "multi-split", ns: "benchdb1.COL-9", details: { before: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey } }, number: 3, of: 3, chunk: { min: { shardkey: 8919152878751157927 }, max: { shardkey: MaxKey }, lastmod: Timestamp 3000|7, lastmodEpoch: ObjectId('5499a93800df4929841903c6') } } } 2014-12-23T17:43:37.492+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:37-5499a9c99a8f659f34bfb9be", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356617492), what: "moveChunk.commit", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:37.573+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:37.573+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:43:37.574+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-4 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 23124210 2014-12-23T17:43:37.574+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-4 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:43:37.574+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:37.677+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-9/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:37.677+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-9", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6005881697089147565 }, { shardkey: 8919152878751157927 } ], shardId: "benchdb1.COL-9-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a93800df4929841903c6') } ntoreturn:1 keyUpdates:0 reslen:37 1030ms 2014-12-23T17:43:37.704+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-9 { : 3074457345618258602 } -->> { : 6005881697089147565 } 2014-12-23T17:43:37.714+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-9 { : 6005881697089147565 } -->> { : 8919152878751157927 } 2014-12-23T17:43:37.724+0000 I SHARDING [conn4] request split points lookup for chunk benchdb1.COL-9 { : 8919152878751157927 } -->> { : MaxKey } 2014-12-23T17:43:37.779+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:37.779+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:37-5499a9c99a8f659f34bfb9bf", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356617779), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 565, step 3 of 6: 1, step 4 of 6: 16, step 5 of 6: 438, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:43:37.861+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } ntoreturn:1 keyUpdates:0 reslen:37 1311ms 2014-12-23T17:43:37.867+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:43:37.867+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } 2014-12-23T17:43:38.208+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9c99a8f659f34bfb9c0 2014-12-23T17:43:38.208+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:38-5499a9ca9a8f659f34bfb9c1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356618208), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:38.283+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 2|0||5499a9c500df492984190422, current metadata version is 2|0||5499a9c500df492984190422 2014-12-23T17:43:38.284+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-4 from shard version 2|0||5499a9c500df492984190422 to shard version 2|1||5499a9c500df492984190422 2014-12-23T17:43:38.284+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a9c500df492984190422, took 0ms 2014-12-23T17:43:38.284+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a9c500df492984190422 2014-12-23T17:43:38.285+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:43:38.285+0000 W SHARDING [conn1] moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration 2014-12-23T17:43:38.285+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:38.344+0000 I INDEX [conn2] build index on: benchdb1.COL-30 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-30" } 2014-12-23T17:43:38.344+0000 I INDEX [conn2] building index using bulk method 2014-12-23T17:43:38.345+0000 I INDEX [conn2] build index done. scanned 0 total records. 0 secs 2014-12-23T17:43:38.488+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:38.488+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:38-5499a9ca9a8f659f34bfb9c3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356618488), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 417, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration" } } 2014-12-23T17:43:38.546+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } ntoreturn:1 keyUpdates:0 reslen:309 679ms 2014-12-23T17:43:38.848+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c500df492984190422') } 2014-12-23T17:43:38.856+0000 I SHARDING [conn2] remotely refreshing metadata for benchdb1.COL-30 with requested shard version 1|2||5499a9c700df492984190425, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:43:38.857+0000 I SHARDING [conn2] collection benchdb1.COL-30 was previously unsharded, new metadata loaded with shard version 1|2||5499a9c700df492984190425 2014-12-23T17:43:38.857+0000 I SHARDING [conn2] collection version was loaded at version 1|2||5499a9c700df492984190425, took 0ms 2014-12-23T17:43:38.932+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:38.932+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:43:39.199+0000 I SHARDING [conn10] request split points lookup for chunk benchdb1.COL-9 { : 8919152878751157927 } -->> { : MaxKey } 2014-12-23T17:43:39.388+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9ca9a8f659f34bfb9c4 2014-12-23T17:43:39.388+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 2|1||5499a9c500df492984190422, current metadata version is 2|1||5499a9c500df492984190422 2014-12-23T17:43:39.389+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-4 already up to date (shard version : 2|1||5499a9c500df492984190422, took 0ms) 2014-12-23T17:43:39.389+0000 I SHARDING [conn1] splitChunk accepted at version 2|1||5499a9c500df492984190422 2014-12-23T17:43:39.489+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9ca9a8f659f34bfb9c5 2014-12-23T17:43:39.489+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:39-5499a9cb9a8f659f34bfb9c6", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356619489), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:43:39.576+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|2||5499a9c700df492984190425, current metadata version is 1|2||5499a9c700df492984190425 2014-12-23T17:43:39.577+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|2||5499a9c700df492984190425, took 0ms) 2014-12-23T17:43:39.577+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a9c700df492984190425 2014-12-23T17:43:39.577+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:39.578+0000 W SHARDING [conn4] moveChunk failed to engage TO-shard in the data transfer: migrate already in progress 2014-12-23T17:43:39.578+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:39.677+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:39-5499a9cb9a8f659f34bfb9c7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356619677), what: "split", ns: "benchdb1.COL-4", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('5499a9c500df492984190422') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('5499a9c500df492984190422') } } } 2014-12-23T17:43:39.861+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:39.861+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:39-5499a9cb9a8f659f34bfb9c8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356619861), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 644, to: "shard0001", from: "shard0002", note: "aborted", errmsg: "moveChunk failed to engage TO-shard in the data transfer: migrate already in progress" } } 2014-12-23T17:43:39.932+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:199 1000ms 2014-12-23T17:43:39.936+0000 I SHARDING [conn4] moveChunk waiting for full cleanup after move 2014-12-23T17:43:39.936+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:43:40.033+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:40.033+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c500df492984190422') } ntoreturn:1 keyUpdates:0 reslen:37 1184ms 2014-12-23T17:43:40.411+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-9 { : 3074457345618258602 } -->> { : 6005881697089147565 } 2014-12-23T17:43:40.445+0000 I SHARDING [conn1] request split points lookup for chunk benchdb1.COL-9 { : 6005881697089147565 } -->> { : 8919152878751157927 } 2014-12-23T17:43:40.446+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9cb9a8f659f34bfb9c9 2014-12-23T17:43:40.446+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:40-5499a9cc9a8f659f34bfb9ca", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356620446), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:40.526+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|2||5499a9c700df492984190425, current metadata version is 1|2||5499a9c700df492984190425 2014-12-23T17:43:40.527+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|2||5499a9c700df492984190425, took 0ms) 2014-12-23T17:43:40.527+0000 I SHARDING [conn4] moveChunk request accepted at version 1|2||5499a9c700df492984190425 2014-12-23T17:43:40.527+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:40.527+0000 W SHARDING [conn4] moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration 2014-12-23T17:43:40.527+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:40.689+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:40.689+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:40-5499a9cc9a8f659f34bfb9cb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356620689), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 590, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration" } } 2014-12-23T17:43:41.124+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:309 1187ms 2014-12-23T17:43:41.128+0000 I SHARDING [conn4] received splitChunk request: { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:43:41.352+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c500df492984190422') } 2014-12-23T17:43:41.452+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9cd9a8f659f34bfb9cc 2014-12-23T17:43:41.452+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|2||5499a9c700df492984190425, current metadata version is 1|2||5499a9c700df492984190425 2014-12-23T17:43:41.453+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|2||5499a9c700df492984190425, took 0ms) 2014-12-23T17:43:41.453+0000 I SHARDING [conn4] splitChunk accepted at version 1|2||5499a9c700df492984190425 2014-12-23T17:43:41.695+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:41-5499a9cd9a8f659f34bfb9ce", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356621695), what: "split", ns: "benchdb1.COL-30", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 1000|3, lastmodEpoch: ObjectId('5499a9c700df492984190425') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 1000|4, lastmodEpoch: ObjectId('5499a9c700df492984190425') } } } 2014-12-23T17:43:41.801+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9cd9a8f659f34bfb9cd 2014-12-23T17:43:41.801+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 2|3||5499a9c500df492984190422, current metadata version is 2|3||5499a9c500df492984190422 2014-12-23T17:43:41.803+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-4 from shard version 2|3||5499a9c500df492984190422 to shard version 2|3||5499a9c500df492984190422 2014-12-23T17:43:41.803+0000 I SHARDING [conn1] collection version was loaded at version 2|5||5499a9c500df492984190422, took 1ms 2014-12-23T17:43:41.803+0000 I SHARDING [conn1] splitChunk accepted at version 2|3||5499a9c500df492984190422 2014-12-23T17:43:41.857+0000 I SHARDING [conn10] request split points lookup for chunk benchdb1.COL-9 { : 3074457345618258602 } -->> { : 6005881697089147565 } 2014-12-23T17:43:41.917+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:41.917+0000 I QUERY [conn4] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:37 789ms 2014-12-23T17:43:41.920+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:41.921+0000 I SHARDING [conn10] received splitChunk request: { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: 0 } ], shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:43:41.942+0000 I NETWORK [initandlisten] connection accepted from 10.233.133.124:42583 #12 (12 connections now open) 2014-12-23T17:43:41.942+0000 I SHARDING [conn12] request split points lookup for chunk benchdb1.COL-9 { : 6005881697089147565 } -->> { : 8919152878751157927 } 2014-12-23T17:43:42.020+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:42-5499a9ce9a8f659f34bfb9d1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356622020), what: "split", ns: "benchdb1.COL-4", details: { before: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey } }, left: { min: { shardkey: 3074457345618258602 }, max: { shardkey: 6148914691236517204 }, lastmod: Timestamp 2000|6, lastmodEpoch: ObjectId('5499a9c500df492984190422') }, right: { min: { shardkey: 6148914691236517204 }, max: { shardkey: MaxKey }, lastmod: Timestamp 2000|7, lastmodEpoch: ObjectId('5499a9c500df492984190422') } } } 2014-12-23T17:43:42.413+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:42.413+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-4", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-4-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c500df492984190422') } ntoreturn:1 keyUpdates:0 reslen:37 1061ms 2014-12-23T17:43:42.466+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9cd9a8f659f34bfb9d0 2014-12-23T17:43:42.466+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|4||5499a9c700df492984190425, current metadata version is 1|4||5499a9c700df492984190425 2014-12-23T17:43:42.467+0000 I SHARDING [conn10] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|4||5499a9c700df492984190425, took 0ms) 2014-12-23T17:43:42.467+0000 I SHARDING [conn10] splitChunk accepted at version 1|4||5499a9c700df492984190425 2014-12-23T17:43:42.517+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9cd9a8f659f34bfb9cf 2014-12-23T17:43:42.517+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:42-5499a9ce9a8f659f34bfb9d2", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356622517), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:42.617+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 3|0||5499a9a800df49298419040d, current metadata version is 3|0||5499a9a800df49298419040d 2014-12-23T17:43:42.618+0000 I SHARDING [conn4] updating metadata for benchdb1.COL-25 from shard version 3|0||5499a9a800df49298419040d to shard version 3|1||5499a9a800df49298419040d 2014-12-23T17:43:42.618+0000 I SHARDING [conn4] collection version was loaded at version 3|1||5499a9a800df49298419040d, took 0ms 2014-12-23T17:43:42.618+0000 I SHARDING [conn4] moveChunk request accepted at version 3|1||5499a9a800df49298419040d 2014-12-23T17:43:42.618+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:42.619+0000 W SHARDING [conn4] moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration 2014-12-23T17:43:42.619+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:42.670+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:42-5499a9ce9a8f659f34bfb9d3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356622670), what: "split", ns: "benchdb1.COL-30", details: { before: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 } }, left: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, lastmod: Timestamp 1000|5, lastmodEpoch: ObjectId('5499a9c700df492984190425') }, right: { min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, lastmod: Timestamp 1000|6, lastmodEpoch: ObjectId('5499a9c700df492984190425') } } } 2014-12-23T17:43:42.842+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:42.842+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:42-5499a9ce9a8f659f34bfb9d4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356622842), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 697, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration" } } 2014-12-23T17:43:42.895+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } ntoreturn:1 keyUpdates:0 reslen:309 974ms 2014-12-23T17:43:42.984+0000 I SHARDING [conn4] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:43.064+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:43.064+0000 I QUERY [conn10] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: 0 } ], shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:37 1143ms 2014-12-23T17:43:43.065+0000 I SHARDING [conn10] received splitChunk request: { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:43:43.350+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-0 2014-12-23T17:43:43.452+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a8fb00df4929841903a4, current metadata version is 3|3||5499a8fb00df4929841903a4 2014-12-23T17:43:43.453+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-0, previous version was 0|0||5499a8fb00df4929841903a4, this is a drop 2014-12-23T17:43:43.453+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-0 at shard version 3|3||5499a8fb00df4929841903a4, took 0ms 2014-12-23T17:43:43.531+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9ce9a8f659f34bfb9d5 2014-12-23T17:43:43.531+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:43-5499a9cf9a8f659f34bfb9d7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356623531), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:43.578+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9cf9a8f659f34bfb9d6 2014-12-23T17:43:43.578+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|6||5499a9c700df492984190425, current metadata version is 1|6||5499a9c700df492984190425 2014-12-23T17:43:43.579+0000 I SHARDING [conn10] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|6||5499a9c700df492984190425, took 0ms) 2014-12-23T17:43:43.579+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 2|7||5499a9ae00df492984190412, current metadata version is 2|7||5499a9ae00df492984190412 2014-12-23T17:43:43.579+0000 I SHARDING [conn10] splitChunk accepted at version 1|6||5499a9c700df492984190425 2014-12-23T17:43:43.580+0000 I SHARDING [conn4] metadata of collection benchdb1.COL-26 already up to date (shard version : 2|7||5499a9ae00df492984190412, took 0ms) 2014-12-23T17:43:43.580+0000 I SHARDING [conn4] moveChunk request accepted at version 2|7||5499a9ae00df492984190412 2014-12-23T17:43:43.580+0000 I SHARDING [conn4] moveChunk number of documents: 0 2014-12-23T17:43:43.580+0000 W SHARDING [conn4] moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration 2014-12-23T17:43:43.580+0000 I SHARDING [conn4] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:43.740+0000 I SHARDING [conn4] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:43.740+0000 I SHARDING [conn4] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:43-5499a9cf9a8f659f34bfb9d8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41427", time: new Date(1419356623740), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, step 2 of 6: 595, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because there are still 1 deletes from previous migration" } } 2014-12-23T17:43:43.818+0000 I QUERY [conn4] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } ntoreturn:1 keyUpdates:0 reslen:309 834ms 2014-12-23T17:43:43.860+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:43-5499a9cf9a8f659f34bfb9d9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356623860), what: "split", ns: "benchdb1.COL-30", details: { before: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey } }, left: { min: { shardkey: 3074457345618258602 }, max: { shardkey: 6148914691236517204 }, lastmod: Timestamp 1000|7, lastmodEpoch: ObjectId('5499a9c700df492984190425') }, right: { min: { shardkey: 6148914691236517204 }, max: { shardkey: MaxKey }, lastmod: Timestamp 1000|8, lastmodEpoch: ObjectId('5499a9c700df492984190425') } } } 2014-12-23T17:43:44.289+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:44.290+0000 I QUERY [conn10] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-30", keyPattern: { shardkey: "hashed" }, min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", splitKeys: [ { shardkey: 6148914691236517204 } ], shardId: "benchdb1.COL-30-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:37 1224ms 2014-12-23T17:43:44.548+0000 I COMMAND [conn10] CMD: drop benchdb1.COL-1 2014-12-23T17:43:44.654+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a90200df4929841903a9, current metadata version is 3|3||5499a90200df4929841903a9 2014-12-23T17:43:44.655+0000 W SHARDING [conn10] no chunks found when reloading benchdb1.COL-1, previous version was 0|0||5499a90200df4929841903a9, this is a drop 2014-12-23T17:43:44.655+0000 I SHARDING [conn10] dropping metadata for benchdb1.COL-1 at shard version 3|3||5499a90200df4929841903a9, took 0ms 2014-12-23T17:43:45.656+0000 I COMMAND [conn10] CMD: drop benchdb1.COL-10 2014-12-23T17:43:45.730+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-10 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a93900df4929841903ca, current metadata version is 3|3||5499a93900df4929841903ca 2014-12-23T17:43:45.738+0000 W SHARDING [conn10] no chunks found when reloading benchdb1.COL-10, previous version was 0|0||5499a93900df4929841903ca, this is a drop 2014-12-23T17:43:45.738+0000 I SHARDING [conn10] dropping metadata for benchdb1.COL-10 at shard version 3|3||5499a93900df4929841903ca, took 8ms 2014-12-23T17:43:46.515+0000 I COMMAND [conn10] CMD: drop benchdb1.COL-11 2014-12-23T17:43:46.587+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-11 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a94100df4929841903d0, current metadata version is 3|3||5499a94100df4929841903d0 2014-12-23T17:43:46.588+0000 W SHARDING [conn10] no chunks found when reloading benchdb1.COL-11, previous version was 0|0||5499a94100df4929841903d0, this is a drop 2014-12-23T17:43:46.588+0000 I SHARDING [conn10] dropping metadata for benchdb1.COL-11 at shard version 3|3||5499a94100df4929841903d0, took 0ms 2014-12-23T17:43:47.154+0000 I SHARDING [conn10] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } 2014-12-23T17:43:47.362+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-12 2014-12-23T17:43:47.449+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-12 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a94800df4929841903d4, current metadata version is 3|3||5499a94800df4929841903d4 2014-12-23T17:43:47.449+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-12, previous version was 0|0||5499a94800df4929841903d4, this is a drop 2014-12-23T17:43:47.449+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-12 at shard version 3|3||5499a94800df4929841903d4, took 0ms 2014-12-23T17:43:47.660+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9d39a8f659f34bfb9da 2014-12-23T17:43:47.660+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:47-5499a9d39a8f659f34bfb9db", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356627660), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:47.726+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 2|7||5499a9c500df492984190422, current metadata version is 2|7||5499a9c500df492984190422 2014-12-23T17:43:47.727+0000 I SHARDING [conn10] metadata of collection benchdb1.COL-4 already up to date (shard version : 2|7||5499a9c500df492984190422, took 0ms) 2014-12-23T17:43:47.727+0000 I SHARDING [conn10] moveChunk request accepted at version 2|7||5499a9c500df492984190422 2014-12-23T17:43:47.742+0000 I SHARDING [conn10] moveChunk number of documents: 0 2014-12-23T17:43:47.744+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:47.747+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:47.751+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:47.760+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:47.776+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:47.777+0000 I SHARDING [conn10] About to check if it is safe to enter critical section 2014-12-23T17:43:47.777+0000 I SHARDING [conn10] About to enter migrate critical section 2014-12-23T17:43:47.784+0000 I SHARDING [conn10] moveChunk setting version to: 3|0||5499a9c500df492984190422 2014-12-23T17:43:47.890+0000 I SHARDING [conn10] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:47.890+0000 I SHARDING [conn10] moveChunk updating self version to: 3|1||5499a9c500df492984190422 through { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-4' 2014-12-23T17:43:48.192+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:48-5499a9d49a8f659f34bfb9dc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356628192), what: "moveChunk.commit", ns: "benchdb1.COL-4", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:48.276+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:48.290+0000 I SHARDING [conn10] forking for cleanup of chunk data 2014-12-23T17:43:48.290+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:48.290+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-4 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 }, with opId: 3 2014-12-23T17:43:48.290+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-4 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 } 2014-12-23T17:43:48.434+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-13 2014-12-23T17:43:48.494+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:48.494+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:48-5499a9d49a8f659f34bfb9dd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356628494), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, step 2 of 6: 573, step 3 of 6: 15, step 4 of 6: 33, step 5 of 6: 513, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:48.504+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-13 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a94f00df4929841903d8, current metadata version is 3|3||5499a94f00df4929841903d8 2014-12-23T17:43:48.505+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-13, previous version was 0|0||5499a94f00df4929841903d8, this is a drop 2014-12-23T17:43:48.505+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-13 at shard version 3|3||5499a94f00df4929841903d8, took 0ms 2014-12-23T17:43:48.550+0000 I QUERY [conn10] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } ntoreturn:1 keyUpdates:0 reslen:37 1397ms 2014-12-23T17:43:48.552+0000 I SHARDING [conn10] received moveChunk request: { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } 2014-12-23T17:43:49.106+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9d49a8f659f34bfb9de 2014-12-23T17:43:49.106+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:49-5499a9d59a8f659f34bfb9df", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356629106), what: "moveChunk.start", ns: "benchdb1.COL-25", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:49.253+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-25 based on current shard version 3|1||5499a9a800df49298419040d, current metadata version is 3|1||5499a9a800df49298419040d 2014-12-23T17:43:49.254+0000 I SHARDING [conn10] metadata of collection benchdb1.COL-25 already up to date (shard version : 3|1||5499a9a800df49298419040d, took 0ms) 2014-12-23T17:43:49.254+0000 I SHARDING [conn10] moveChunk request accepted at version 3|1||5499a9a800df49298419040d 2014-12-23T17:43:49.264+0000 I SHARDING [conn10] moveChunk number of documents: 0 2014-12-23T17:43:49.267+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:49.269+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:49.269+0000 I SHARDING [conn10] About to check if it is safe to enter critical section 2014-12-23T17:43:49.270+0000 I SHARDING [conn10] About to enter migrate critical section 2014-12-23T17:43:49.270+0000 I SHARDING [conn10] moveChunk setting version to: 4|0||5499a9a800df49298419040d 2014-12-23T17:43:49.331+0000 I SHARDING [conn10] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:49.331+0000 I SHARDING [conn10] moveChunk updating self version to: 4|1||5499a9a800df49298419040d through { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } for collection 'benchdb1.COL-25' 2014-12-23T17:43:49.592+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:49-5499a9d59a8f659f34bfb9e0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356629592), what: "moveChunk.commit", ns: "benchdb1.COL-25", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:49.665+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:49.668+0000 I SHARDING [conn10] forking for cleanup of chunk data 2014-12-23T17:43:49.668+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:49.668+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-25 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T17:43:49.668+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-25 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } 2014-12-23T17:43:49.719+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-14 2014-12-23T17:43:49.783+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-14 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a95900df4929841903dd, current metadata version is 3|3||5499a95900df4929841903dd 2014-12-23T17:43:49.784+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-14, previous version was 0|0||5499a95900df4929841903dd, this is a drop 2014-12-23T17:43:49.784+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-14 at shard version 3|3||5499a95900df4929841903dd, took 0ms 2014-12-23T17:43:49.848+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-25/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:49.848+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:49-5499a9d59a8f659f34bfb9e1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356629848), what: "moveChunk.from", ns: "benchdb1.COL-25", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 701, step 3 of 6: 11, step 4 of 6: 4, step 5 of 6: 398, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:49.900+0000 I QUERY [conn10] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-25", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-25-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9a800df49298419040d') } ntoreturn:1 keyUpdates:0 reslen:37 1348ms 2014-12-23T17:43:49.902+0000 I SHARDING [conn10] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:50.422+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9d59a8f659f34bfb9e2 2014-12-23T17:43:50.422+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:50-5499a9d69a8f659f34bfb9e3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356630422), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:50.490+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 2|7||5499a9ae00df492984190412, current metadata version is 2|7||5499a9ae00df492984190412 2014-12-23T17:43:50.491+0000 I SHARDING [conn10] metadata of collection benchdb1.COL-26 already up to date (shard version : 2|7||5499a9ae00df492984190412, took 0ms) 2014-12-23T17:43:50.491+0000 I SHARDING [conn10] moveChunk request accepted at version 2|7||5499a9ae00df492984190412 2014-12-23T17:43:50.491+0000 I SHARDING [conn10] moveChunk number of documents: 0 2014-12-23T17:43:50.494+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:50.496+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:50.500+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:50.509+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:50.525+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:50.525+0000 I SHARDING [conn10] About to check if it is safe to enter critical section 2014-12-23T17:43:50.526+0000 I SHARDING [conn10] About to enter migrate critical section 2014-12-23T17:43:50.526+0000 I SHARDING [conn10] moveChunk setting version to: 3|0||5499a9ae00df492984190412 2014-12-23T17:43:50.583+0000 I SHARDING [conn10] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:50.583+0000 I SHARDING [conn10] moveChunk updating self version to: 3|1||5499a9ae00df492984190412 through { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-26' 2014-12-23T17:43:50.737+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-15 2014-12-23T17:43:50.816+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-15 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a96000df4929841903e3, current metadata version is 3|3||5499a96000df4929841903e3 2014-12-23T17:43:50.817+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-15, previous version was 0|0||5499a96000df4929841903e3, this is a drop 2014-12-23T17:43:50.817+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-15 at shard version 3|3||5499a96000df4929841903e3, took 0ms 2014-12-23T17:43:50.862+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:50-5499a9d69a8f659f34bfb9e4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356630862), what: "moveChunk.commit", ns: "benchdb1.COL-26", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:50.925+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:50.927+0000 I SHARDING [conn10] forking for cleanup of chunk data 2014-12-23T17:43:50.927+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:50.927+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-26 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 }, with opId: 3 2014-12-23T17:43:50.927+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-26 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 } 2014-12-23T17:43:51.206+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:51.206+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:51-5499a9d79a8f659f34bfb9e5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356631206), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, step 2 of 6: 589, step 3 of 6: 1, step 4 of 6: 33, step 5 of 6: 401, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:51.276+0000 I QUERY [conn10] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } ntoreturn:1 keyUpdates:0 reslen:37 1374ms 2014-12-23T17:43:51.279+0000 I SHARDING [conn10] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:43:51.771+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9d79a8f659f34bfb9e6 2014-12-23T17:43:51.771+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:51-5499a9d79a8f659f34bfb9e7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356631771), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:51.833+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 1|8||5499a9c700df492984190425, current metadata version is 1|8||5499a9c700df492984190425 2014-12-23T17:43:51.834+0000 I SHARDING [conn10] metadata of collection benchdb1.COL-30 already up to date (shard version : 1|8||5499a9c700df492984190425, took 0ms) 2014-12-23T17:43:51.834+0000 I SHARDING [conn10] moveChunk request accepted at version 1|8||5499a9c700df492984190425 2014-12-23T17:43:51.836+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-16 2014-12-23T17:43:51.847+0000 I SHARDING [conn10] moveChunk number of documents: 0 2014-12-23T17:43:51.849+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:51.851+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:51.856+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:51.864+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:51.881+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:51.881+0000 I SHARDING [conn10] About to check if it is safe to enter critical section 2014-12-23T17:43:51.881+0000 I SHARDING [conn10] About to enter migrate critical section 2014-12-23T17:43:51.881+0000 I SHARDING [conn10] moveChunk setting version to: 2|0||5499a9c700df492984190425 2014-12-23T17:43:51.911+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-16 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a96700df4929841903e7, current metadata version is 3|3||5499a96700df4929841903e7 2014-12-23T17:43:51.911+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-16, previous version was 0|0||5499a96700df4929841903e7, this is a drop 2014-12-23T17:43:51.911+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-16 at shard version 3|3||5499a96700df4929841903e7, took 0ms 2014-12-23T17:43:51.972+0000 I SHARDING [conn10] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:51.972+0000 I SHARDING [conn10] moveChunk updating self version to: 2|1||5499a9c700df492984190425 through { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-30' 2014-12-23T17:43:52.268+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:52-5499a9d89a8f659f34bfb9e8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356632268), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:52.353+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:52.353+0000 I SHARDING [conn10] forking for cleanup of chunk data 2014-12-23T17:43:52.353+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:52.353+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-30 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 }, with opId: 3 2014-12-23T17:43:52.354+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: MinKey } -> { shardkey: -6148914691236517204 } 2014-12-23T17:43:52.610+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:52.610+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:52-5499a9d89a8f659f34bfb9e9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356632610), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, step 1 of 6: 0, step 2 of 6: 554, step 3 of 6: 13, step 4 of 6: 33, step 5 of 6: 472, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:52.668+0000 I QUERY [conn10] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:37 1389ms 2014-12-23T17:43:52.872+0000 I COMMAND [conn10] CMD: drop benchdb1.COL-17 2014-12-23T17:43:52.961+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-17 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a96e00df4929841903ec, current metadata version is 3|3||5499a96e00df4929841903ec 2014-12-23T17:43:52.962+0000 W SHARDING [conn10] no chunks found when reloading benchdb1.COL-17, previous version was 0|0||5499a96e00df4929841903ec, this is a drop 2014-12-23T17:43:52.962+0000 I SHARDING [conn10] dropping metadata for benchdb1.COL-17 at shard version 3|3||5499a96e00df4929841903ec, took 0ms 2014-12-23T17:43:53.181+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:43:52 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:43:54.067+0000 I COMMAND [conn10] CMD: drop benchdb1.COL-18 2014-12-23T17:43:54.955+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-18 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a97500df4929841903ef, current metadata version is 3|3||5499a97500df4929841903ef 2014-12-23T17:43:54.956+0000 W SHARDING [conn10] no chunks found when reloading benchdb1.COL-18, previous version was 0|0||5499a97500df4929841903ef, this is a drop 2014-12-23T17:43:54.956+0000 I SHARDING [conn10] dropping metadata for benchdb1.COL-18 at shard version 3|3||5499a97500df4929841903ef, took 0ms 2014-12-23T17:43:55.245+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-1", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 114ms 2014-12-23T17:43:55.584+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-1", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 119ms 2014-12-23T17:43:55.694+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-1", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 104ms 2014-12-23T17:43:55.784+0000 I COMMAND [conn10] CMD: drop benchdb1.COL-19 2014-12-23T17:43:55.861+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-19 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a97c00df4929841903f4, current metadata version is 3|3||5499a97c00df4929841903f4 2014-12-23T17:43:55.871+0000 W SHARDING [conn10] no chunks found when reloading benchdb1.COL-19, previous version was 0|0||5499a97c00df4929841903f4, this is a drop 2014-12-23T17:43:55.871+0000 I SHARDING [conn10] dropping metadata for benchdb1.COL-19 at shard version 3|3||5499a97c00df4929841903f4, took 9ms 2014-12-23T17:43:55.918+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-1", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 124ms 2014-12-23T17:43:56.063+0000 I QUERY [conn2] command benchdb1.$cmd command: insert { insert: "COL-1", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 131ms 2014-12-23T17:43:56.371+0000 I SHARDING [conn10] received moveChunk request: { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } 2014-12-23T17:43:56.731+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-2 2014-12-23T17:43:56.931+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9dc9a8f659f34bfb9ea 2014-12-23T17:43:56.931+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:56-5499a9dc9a8f659f34bfb9eb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356636931), what: "moveChunk.start", ns: "benchdb1.COL-4", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:57.160+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-2 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a90a00df4929841903ac, current metadata version is 3|3||5499a90a00df4929841903ac 2014-12-23T17:43:57.161+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-2, previous version was 0|0||5499a90a00df4929841903ac, this is a drop 2014-12-23T17:43:57.161+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-2 at shard version 3|3||5499a90a00df4929841903ac, took 0ms 2014-12-23T17:43:57.217+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-4 based on current shard version 3|0||5499a9c500df492984190422, current metadata version is 3|0||5499a9c500df492984190422 2014-12-23T17:43:57.218+0000 I SHARDING [conn10] updating metadata for benchdb1.COL-4 from shard version 3|0||5499a9c500df492984190422 to shard version 3|1||5499a9c500df492984190422 2014-12-23T17:43:57.218+0000 I SHARDING [conn10] collection version was loaded at version 3|1||5499a9c500df492984190422, took 0ms 2014-12-23T17:43:57.218+0000 I SHARDING [conn10] moveChunk request accepted at version 3|1||5499a9c500df492984190422 2014-12-23T17:43:57.232+0000 I SHARDING [conn10] moveChunk number of documents: 0 2014-12-23T17:43:57.234+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:57.237+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:57.239+0000 I SHARDING [conn10] About to check if it is safe to enter critical section 2014-12-23T17:43:57.240+0000 I SHARDING [conn10] About to enter migrate critical section 2014-12-23T17:43:57.240+0000 I SHARDING [conn10] moveChunk setting version to: 4|0||5499a9c500df492984190422 2014-12-23T17:43:57.310+0000 I SHARDING [conn10] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:57.311+0000 I SHARDING [conn10] moveChunk updating self version to: 4|1||5499a9c500df492984190422 through { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } for collection 'benchdb1.COL-4' 2014-12-23T17:43:57.569+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:57-5499a9dd9a8f659f34bfb9ec", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356637569), what: "moveChunk.commit", ns: "benchdb1.COL-4", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:57.678+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:57.685+0000 I SHARDING [conn10] forking for cleanup of chunk data 2014-12-23T17:43:57.685+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:57.685+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-4 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T17:43:57.685+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-4 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } 2014-12-23T17:43:57.986+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-4/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:58.016+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:58-5499a9de9a8f659f34bfb9ed", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356638015), what: "moveChunk.from", ns: "benchdb1.COL-4", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 846, step 3 of 6: 14, step 4 of 6: 6, step 5 of 6: 445, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:58.086+0000 I QUERY [conn10] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-4", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-4-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c500df492984190422') } ntoreturn:1 keyUpdates:0 reslen:37 1714ms 2014-12-23T17:43:58.088+0000 I SHARDING [conn10] received moveChunk request: { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } 2014-12-23T17:43:58.190+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-20 2014-12-23T17:43:58.306+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-20 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a98300df4929841903f8, current metadata version is 3|3||5499a98300df4929841903f8 2014-12-23T17:43:58.306+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-20, previous version was 0|0||5499a98300df4929841903f8, this is a drop 2014-12-23T17:43:58.306+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-20 at shard version 3|3||5499a98300df4929841903f8, took 0ms 2014-12-23T17:43:58.631+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9de9a8f659f34bfb9ee 2014-12-23T17:43:58.631+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:58-5499a9de9a8f659f34bfb9ef", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356638631), what: "moveChunk.start", ns: "benchdb1.COL-26", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:43:58.688+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-26 based on current shard version 3|0||5499a9ae00df492984190412, current metadata version is 3|0||5499a9ae00df492984190412 2014-12-23T17:43:58.689+0000 I SHARDING [conn10] updating metadata for benchdb1.COL-26 from shard version 3|0||5499a9ae00df492984190412 to shard version 3|1||5499a9ae00df492984190412 2014-12-23T17:43:58.689+0000 I SHARDING [conn10] collection version was loaded at version 3|1||5499a9ae00df492984190412, took 0ms 2014-12-23T17:43:58.689+0000 I SHARDING [conn10] moveChunk request accepted at version 3|1||5499a9ae00df492984190412 2014-12-23T17:43:58.698+0000 I SHARDING [conn10] moveChunk number of documents: 0 2014-12-23T17:43:58.701+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:58.703+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:43:58.703+0000 I SHARDING [conn10] About to check if it is safe to enter critical section 2014-12-23T17:43:58.704+0000 I SHARDING [conn10] About to enter migrate critical section 2014-12-23T17:43:58.704+0000 I SHARDING [conn10] moveChunk setting version to: 4|0||5499a9ae00df492984190412 2014-12-23T17:43:58.783+0000 I SHARDING [conn10] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:43:58.783+0000 I SHARDING [conn10] moveChunk updating self version to: 4|1||5499a9ae00df492984190412 through { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } for collection 'benchdb1.COL-26' 2014-12-23T17:43:59.084+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:59-5499a9df9a8f659f34bfb9f0", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356639083), what: "moveChunk.commit", ns: "benchdb1.COL-26", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:43:59.159+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:59.159+0000 I SHARDING [conn10] forking for cleanup of chunk data 2014-12-23T17:43:59.159+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:43:59.159+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-26 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T17:43:59.159+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-26 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } 2014-12-23T17:43:59.369+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-21 2014-12-23T17:43:59.441+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-26/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:43:59.441+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:59-5499a9df9a8f659f34bfb9f1", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356639441), what: "moveChunk.from", ns: "benchdb1.COL-26", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 601, step 3 of 6: 10, step 4 of 6: 4, step 5 of 6: 455, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:43:59.456+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-21 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a98b00df4929841903fc, current metadata version is 3|3||5499a98b00df4929841903fc 2014-12-23T17:43:59.457+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-21, previous version was 0|0||5499a98b00df4929841903fc, this is a drop 2014-12-23T17:43:59.457+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-21 at shard version 3|3||5499a98b00df4929841903fc, took 0ms 2014-12-23T17:43:59.506+0000 I QUERY [conn10] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-26", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-26-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9ae00df492984190412') } ntoreturn:1 keyUpdates:0 reslen:37 1419ms 2014-12-23T17:43:59.508+0000 I SHARDING [conn10] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:43:59.996+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9df9a8f659f34bfb9f2 2014-12-23T17:43:59.996+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:43:59-5499a9df9a8f659f34bfb9f3", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356639996), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:44:00.056+0000 I SHARDING [conn10] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 2|0||5499a9c700df492984190425, current metadata version is 2|0||5499a9c700df492984190425 2014-12-23T17:44:00.057+0000 I SHARDING [conn10] updating metadata for benchdb1.COL-30 from shard version 2|0||5499a9c700df492984190425 to shard version 2|1||5499a9c700df492984190425 2014-12-23T17:44:00.057+0000 I SHARDING [conn10] collection version was loaded at version 2|1||5499a9c700df492984190425, took 0ms 2014-12-23T17:44:00.057+0000 I SHARDING [conn10] moveChunk request accepted at version 2|1||5499a9c700df492984190425 2014-12-23T17:44:00.071+0000 I SHARDING [conn10] moveChunk number of documents: 0 2014-12-23T17:44:00.074+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:00.077+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:00.081+0000 I SHARDING [conn10] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:00.081+0000 I SHARDING [conn10] About to check if it is safe to enter critical section 2014-12-23T17:44:00.082+0000 I SHARDING [conn10] About to enter migrate critical section 2014-12-23T17:44:00.082+0000 I SHARDING [conn10] moveChunk setting version to: 3|0||5499a9c700df492984190425 2014-12-23T17:44:00.157+0000 I SHARDING [conn10] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:44:00.157+0000 I SHARDING [conn10] moveChunk updating self version to: 3|1||5499a9c700df492984190425 through { shardkey: -3074457345618258602 } -> { shardkey: 0 } for collection 'benchdb1.COL-30' 2014-12-23T17:44:00.462+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:00-5499a9e09a8f659f34bfb9f4", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356640462), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:44:00.465+0000 I COMMAND [conn4] CMD: drop benchdb1.COL-22 2014-12-23T17:44:00.549+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:00.549+0000 I SHARDING [conn10] forking for cleanup of chunk data 2014-12-23T17:44:00.549+0000 I SHARDING [conn10] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:00.549+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-30 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 }, with opId: 3 2014-12-23T17:44:00.549+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: -6148914691236517204 } -> { shardkey: -3074457345618258602 } 2014-12-23T17:44:00.597+0000 I SHARDING [conn4] remotely refreshing metadata for benchdb1.COL-22 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a99100df492984190401, current metadata version is 3|3||5499a99100df492984190401 2014-12-23T17:44:00.597+0000 W SHARDING [conn4] no chunks found when reloading benchdb1.COL-22, previous version was 0|0||5499a99100df492984190401, this is a drop 2014-12-23T17:44:00.598+0000 I SHARDING [conn4] dropping metadata for benchdb1.COL-22 at shard version 3|3||5499a99100df492984190401, took 0ms 2014-12-23T17:44:00.804+0000 I SHARDING [conn10] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:00.804+0000 I SHARDING [conn10] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:00-5499a9e09a8f659f34bfb9f5", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42214", time: new Date(1419356640804), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 549, step 3 of 6: 15, step 4 of 6: 8, step 5 of 6: 467, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:44:00.865+0000 I QUERY [conn10] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-6148914691236517204", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:37 1357ms 2014-12-23T17:44:01.609+0000 I COMMAND [conn10] CMD: drop benchdb1.COL-23 2014-12-23T17:44:01.957+0000 I SHARDING [conn12] remotely refreshing metadata for benchdb1.COL-23 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a99a00df492984190405, current metadata version is 3|3||5499a99a00df492984190405 2014-12-23T17:44:01.958+0000 W SHARDING [conn12] no chunks found when reloading benchdb1.COL-23, previous version was 0|0||5499a99a00df492984190405, this is a drop 2014-12-23T17:44:01.958+0000 I SHARDING [conn12] dropping metadata for benchdb1.COL-23 at shard version 3|3||5499a99a00df492984190405, took 0ms 2014-12-23T17:44:03.057+0000 I COMMAND [conn12] CMD: drop benchdb1.COL-24 2014-12-23T17:44:03.157+0000 I SHARDING [conn12] remotely refreshing metadata for benchdb1.COL-24 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a9a100df492984190409, current metadata version is 3|3||5499a9a100df492984190409 2014-12-23T17:44:03.159+0000 W SHARDING [conn12] no chunks found when reloading benchdb1.COL-24, previous version was 0|0||5499a9a100df492984190409, this is a drop 2014-12-23T17:44:03.159+0000 I SHARDING [conn12] dropping metadata for benchdb1.COL-24 at shard version 3|3||5499a9a100df492984190409, took 1ms 2014-12-23T17:44:03.699+0000 I SHARDING [conn12] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:44:03.912+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-25 2014-12-23T17:44:04.001+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-25 with requested shard version 0|0||000000000000000000000000, current shard version is 4|0||5499a9a800df49298419040d, current metadata version is 4|0||5499a9a800df49298419040d 2014-12-23T17:44:04.002+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-25, previous version was 0|0||5499a9a800df49298419040d, this is a drop 2014-12-23T17:44:04.002+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-25 at shard version 4|0||5499a9a800df49298419040d, took 0ms 2014-12-23T17:44:04.223+0000 I SHARDING [conn12] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9e39a8f659f34bfb9f6 2014-12-23T17:44:04.224+0000 I SHARDING [conn12] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:04-5499a9e49a8f659f34bfb9f7", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42583", time: new Date(1419356644224), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:44:04.317+0000 I SHARDING [conn12] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 3|0||5499a9c700df492984190425, current metadata version is 3|0||5499a9c700df492984190425 2014-12-23T17:44:04.318+0000 I SHARDING [conn12] updating metadata for benchdb1.COL-30 from shard version 3|0||5499a9c700df492984190425 to shard version 3|1||5499a9c700df492984190425 2014-12-23T17:44:04.318+0000 I SHARDING [conn12] collection version was loaded at version 3|1||5499a9c700df492984190425, took 0ms 2014-12-23T17:44:04.318+0000 I SHARDING [conn12] moveChunk request accepted at version 3|1||5499a9c700df492984190425 2014-12-23T17:44:04.325+0000 I SHARDING [conn12] moveChunk number of documents: 0 2014-12-23T17:44:04.327+0000 I SHARDING [conn12] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:04.330+0000 I SHARDING [conn12] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:04.330+0000 I SHARDING [conn12] About to check if it is safe to enter critical section 2014-12-23T17:44:04.331+0000 I SHARDING [conn12] About to enter migrate critical section 2014-12-23T17:44:04.331+0000 I SHARDING [conn12] moveChunk setting version to: 4|0||5499a9c700df492984190425 2014-12-23T17:44:04.395+0000 I SHARDING [conn12] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:44:04.395+0000 I SHARDING [conn12] moveChunk updating self version to: 4|1||5499a9c700df492984190425 through { shardkey: 0 } -> { shardkey: 3074457345618258602 } for collection 'benchdb1.COL-30' 2014-12-23T17:44:04.648+0000 I SHARDING [conn12] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:04-5499a9e49a8f659f34bfb9f8", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42583", time: new Date(1419356644648), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:44:04.722+0000 I SHARDING [conn12] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:04.735+0000 I SHARDING [conn12] forking for cleanup of chunk data 2014-12-23T17:44:04.735+0000 I SHARDING [conn12] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:04.735+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-30 from { shardkey: -3074457345618258602 } -> { shardkey: 0 }, with opId: 3 2014-12-23T17:44:04.735+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: -3074457345618258602 } -> { shardkey: 0 } 2014-12-23T17:44:04.930+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-26 2014-12-23T17:44:05.009+0000 I SHARDING [conn12] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:05.009+0000 I SHARDING [conn12] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:05-5499a9e59a8f659f34bfb9f9", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42583", time: new Date(1419356645009), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, step 1 of 6: 0, step 2 of 6: 619, step 3 of 6: 7, step 4 of 6: 4, step 5 of 6: 404, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:44:05.015+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-26 with requested shard version 0|0||000000000000000000000000, current shard version is 4|0||5499a9ae00df492984190412, current metadata version is 4|0||5499a9ae00df492984190412 2014-12-23T17:44:05.015+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-26, previous version was 0|0||5499a9ae00df492984190412, this is a drop 2014-12-23T17:44:05.016+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-26 at shard version 4|0||5499a9ae00df492984190412, took 0ms 2014-12-23T17:44:05.080+0000 I QUERY [conn12] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: -3074457345618258602 }, max: { shardkey: 0 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:37 1381ms 2014-12-23T17:44:06.171+0000 I COMMAND [conn12] CMD: drop benchdb1.COL-27 2014-12-23T17:44:06.497+0000 I SHARDING [conn12] remotely refreshing metadata for benchdb1.COL-27 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a9b400df492984190417, current metadata version is 3|3||5499a9b400df492984190417 2014-12-23T17:44:06.498+0000 W SHARDING [conn12] no chunks found when reloading benchdb1.COL-27, previous version was 0|0||5499a9b400df492984190417, this is a drop 2014-12-23T17:44:06.498+0000 I SHARDING [conn12] dropping metadata for benchdb1.COL-27 at shard version 3|3||5499a9b400df492984190417, took 0ms 2014-12-23T17:44:07.176+0000 I COMMAND [conn12] CMD: drop benchdb1.COL-28 2014-12-23T17:44:07.285+0000 I SHARDING [conn12] remotely refreshing metadata for benchdb1.COL-28 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a9ba00df49298419041b, current metadata version is 3|3||5499a9ba00df49298419041b 2014-12-23T17:44:07.286+0000 W SHARDING [conn12] no chunks found when reloading benchdb1.COL-28, previous version was 0|0||5499a9ba00df49298419041b, this is a drop 2014-12-23T17:44:07.286+0000 I SHARDING [conn12] dropping metadata for benchdb1.COL-28 at shard version 3|3||5499a9ba00df49298419041b, took 0ms 2014-12-23T17:44:07.432+0000 I SHARDING [conn12] received moveChunk request: { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_0", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } 2014-12-23T17:44:07.935+0000 I SHARDING [conn12] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9e79a8f659f34bfb9fa 2014-12-23T17:44:07.935+0000 I SHARDING [conn12] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:07-5499a9e79a8f659f34bfb9fb", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42583", time: new Date(1419356647935), what: "moveChunk.start", ns: "benchdb1.COL-30", details: { min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:44:07.996+0000 I SHARDING [conn12] remotely refreshing metadata for benchdb1.COL-30 based on current shard version 4|0||5499a9c700df492984190425, current metadata version is 4|0||5499a9c700df492984190425 2014-12-23T17:44:07.996+0000 I SHARDING [conn12] updating metadata for benchdb1.COL-30 from shard version 4|0||5499a9c700df492984190425 to shard version 4|1||5499a9c700df492984190425 2014-12-23T17:44:07.996+0000 I SHARDING [conn12] collection version was loaded at version 4|1||5499a9c700df492984190425, took 0ms 2014-12-23T17:44:07.996+0000 I SHARDING [conn12] moveChunk request accepted at version 4|1||5499a9c700df492984190425 2014-12-23T17:44:07.997+0000 I SHARDING [conn12] moveChunk number of documents: 0 2014-12-23T17:44:07.999+0000 I SHARDING [conn12] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:08.001+0000 I SHARDING [conn12] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:08.002+0000 I SHARDING [conn12] About to check if it is safe to enter critical section 2014-12-23T17:44:08.002+0000 I SHARDING [conn12] About to enter migrate critical section 2014-12-23T17:44:08.002+0000 I SHARDING [conn12] moveChunk setting version to: 5|0||5499a9c700df492984190425 2014-12-23T17:44:08.062+0000 I SHARDING [conn12] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:44:08.063+0000 I SHARDING [conn12] moveChunk updating self version to: 5|1||5499a9c700df492984190425 through { shardkey: 3074457345618258602 } -> { shardkey: 6148914691236517204 } for collection 'benchdb1.COL-30' 2014-12-23T17:44:08.172+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-29 2014-12-23T17:44:08.255+0000 I SHARDING [conn12] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:08-5499a9e89a8f659f34bfb9fc", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42583", time: new Date(1419356648255), what: "moveChunk.commit", ns: "benchdb1.COL-30", details: { min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:44:08.259+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-29 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a9c100df49298419041f, current metadata version is 3|3||5499a9c100df49298419041f 2014-12-23T17:44:08.260+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-29, previous version was 0|0||5499a9c100df49298419041f, this is a drop 2014-12-23T17:44:08.260+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-29 at shard version 3|3||5499a9c100df49298419041f, took 0ms 2014-12-23T17:44:08.316+0000 I SHARDING [conn12] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:08.316+0000 I SHARDING [conn12] forking for cleanup of chunk data 2014-12-23T17:44:08.316+0000 I SHARDING [conn12] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:08.316+0000 I SHARDING [RangeDeleter] Deleter starting delete for: benchdb1.COL-30 from { shardkey: 0 } -> { shardkey: 3074457345618258602 }, with opId: 3 2014-12-23T17:44:08.316+0000 I SHARDING [RangeDeleter] rangeDeleter deleted 0 documents for benchdb1.COL-30 from { shardkey: 0 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:44:08.557+0000 I SHARDING [conn12] distributed lock 'benchdb1.COL-30/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:08.557+0000 I SHARDING [conn12] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:08-5499a9e89a8f659f34bfb9fd", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42583", time: new Date(1419356648557), what: "moveChunk.from", ns: "benchdb1.COL-30", details: { min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 564, step 3 of 6: 1, step 4 of 6: 3, step 5 of 6: 314, step 6 of 6: 0, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:44:08.617+0000 I QUERY [conn12] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-30", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: 0 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-30-shardkey_0", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9c700df492984190425') } ntoreturn:1 keyUpdates:0 reslen:37 1184ms 2014-12-23T17:44:09.254+0000 I COMMAND [conn12] CMD: drop benchdb1.COL-3 2014-12-23T17:44:09.334+0000 I SHARDING [conn12] remotely refreshing metadata for benchdb1.COL-3 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a91100df4929841903b1, current metadata version is 3|3||5499a91100df4929841903b1 2014-12-23T17:44:09.335+0000 W SHARDING [conn12] no chunks found when reloading benchdb1.COL-3, previous version was 0|0||5499a91100df4929841903b1, this is a drop 2014-12-23T17:44:09.335+0000 I SHARDING [conn12] dropping metadata for benchdb1.COL-3 at shard version 3|3||5499a91100df4929841903b1, took 0ms 2014-12-23T17:44:10.359+0000 I COMMAND [conn12] CMD: drop benchdb1.COL-30 2014-12-23T17:44:10.633+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-30 with requested shard version 0|0||000000000000000000000000, current shard version is 5|0||5499a9c700df492984190425, current metadata version is 5|0||5499a9c700df492984190425 2014-12-23T17:44:10.633+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-30, previous version was 0|0||5499a9c700df492984190425, this is a drop 2014-12-23T17:44:10.633+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-30 at shard version 5|0||5499a9c700df492984190425, took 0ms 2014-12-23T17:44:11.675+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-4 2014-12-23T17:44:11.772+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-4 with requested shard version 0|0||000000000000000000000000, current shard version is 4|0||5499a9c500df492984190422, current metadata version is 4|0||5499a9c500df492984190422 2014-12-23T17:44:11.772+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-4, previous version was 0|0||5499a9c500df492984190422, this is a drop 2014-12-23T17:44:11.772+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-4 at shard version 4|0||5499a9c500df492984190422, took 0ms 2014-12-23T17:44:12.927+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-5 2014-12-23T17:44:13.054+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-5 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a91a00df4929841903b6, current metadata version is 4|1||5499a91a00df4929841903b6 2014-12-23T17:44:13.055+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-5, previous version was 0|0||5499a91a00df4929841903b6, this is a drop 2014-12-23T17:44:13.055+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-5 at shard version 3|3||5499a91a00df4929841903b6, took 0ms 2014-12-23T17:44:14.223+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-6 2014-12-23T17:44:14.608+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-6 with requested shard version 0|0||000000000000000000000000, current shard version is 3|10||5499a92100df4929841903b9, current metadata version is 3|10||5499a92100df4929841903b9 2014-12-23T17:44:14.608+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-6, previous version was 0|0||5499a92100df4929841903b9, this is a drop 2014-12-23T17:44:14.608+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-6 at shard version 3|10||5499a92100df4929841903b9, took 0ms 2014-12-23T17:44:15.734+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-7 2014-12-23T17:44:16.072+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-7 with requested shard version 0|0||000000000000000000000000, current shard version is 3|10||5499a92900df4929841903be, current metadata version is 3|10||5499a92900df4929841903be 2014-12-23T17:44:16.072+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-7, previous version was 0|0||5499a92900df4929841903be, this is a drop 2014-12-23T17:44:16.073+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-7 at shard version 3|10||5499a92900df4929841903be, took 0ms 2014-12-23T17:44:17.238+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-8 2014-12-23T17:44:17.344+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-8 with requested shard version 0|0||000000000000000000000000, current shard version is 3|10||5499a93000df4929841903c3, current metadata version is 3|10||5499a93000df4929841903c3 2014-12-23T17:44:17.344+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-8, previous version was 0|0||5499a93000df4929841903c3, this is a drop 2014-12-23T17:44:17.345+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-8 at shard version 3|10||5499a93000df4929841903c3, took 0ms 2014-12-23T17:44:18.512+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-9 2014-12-23T17:44:18.628+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-9 with requested shard version 0|0||000000000000000000000000, current shard version is 3|7||5499a93800df4929841903c6, current metadata version is 3|7||5499a93800df4929841903c6 2014-12-23T17:44:18.629+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-9, previous version was 0|0||5499a93800df4929841903c6, this is a drop 2014-12-23T17:44:18.629+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-9 at shard version 3|7||5499a93800df4929841903c6, took 0ms 2014-12-23T17:44:19.057+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:44:19.709+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:44:19.709+0000 I QUERY [conn6] command benchdb1.$cmd command: dropDatabase { dropDatabase: 1 } ntoreturn:1 keyUpdates:0 reslen:59 659ms 2014-12-23T17:44:20.046+0000 I WRITE [conn11] insert benchdb1.COL-13 query: { _id: ObjectId('5498b3590315362bd8e53be4'), shardkey: "4asVpkv9ByXNUaTJmwt1H1o1", payload: "64HqzReZIgtaGA5B3jjgfDojR9qDjskBIq8mtijeBGzqw7sHAaQbjyOgfzuC51YeDnK15PQQoJXmojKjdFVlr5bwxNhpqqQiXFd8KvsBLgXgzbVCZHqI0tYMLHvoOCIN8zIrxHCIBKDjqFehtn1zxn..." } ninserted:1 keyUpdates:0 984ms 2014-12-23T17:44:20.061+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:44:20.061+0000 I QUERY [conn11] command benchdb1.$cmd command: insert { insert: "COL-13", documents: 1000, writeConcern: { w: 1 }, ordered: false, metadata: { shardName: "shard0002", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 reslen:40 1000ms 2014-12-23T17:44:20.062+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:44:20.062+0000 I QUERY [conn6] command benchdb1.$cmd command: dropDatabase { dropDatabase: 1 } ntoreturn:1 keyUpdates:0 reslen:59 275ms 2014-12-23T17:44:20.172+0000 I INDEX [conn9] build index on: benchdb1.COL-0 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-0" } 2014-12-23T17:44:20.172+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:44:20.173+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:44:20.532+0000 I SHARDING [conn11] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 1|2||5499a9f100df492984190492, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:44:20.533+0000 I SHARDING [conn11] collection benchdb1.COL-0 was previously unsharded, new metadata loaded with shard version 1|2||5499a9f100df492984190492 2014-12-23T17:44:20.533+0000 I SHARDING [conn11] collection version was loaded at version 1|2||5499a9f100df492984190492, took 0ms 2014-12-23T17:44:20.580+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:44:20.580+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f100df492984190492') } 2014-12-23T17:44:21.187+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9f49a8f659f34bfb9ff 2014-12-23T17:44:21.187+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:21-5499a9f59a8f659f34bfba00", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356661187), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:44:21.259+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 1|2||5499a9f100df492984190492, current metadata version is 1|2||5499a9f100df492984190492 2014-12-23T17:44:21.260+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-0 already up to date (shard version : 1|2||5499a9f100df492984190492, took 0ms) 2014-12-23T17:44:21.260+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a9f100df492984190492 2014-12-23T17:44:21.260+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:44:21.263+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:21.265+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:21.270+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:21.270+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:44:21.270+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:44:21.270+0000 I SHARDING [conn1] moveChunk setting version to: 2|0||5499a9f100df492984190492 2014-12-23T17:44:21.329+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:44:21.329+0000 I SHARDING [conn1] moveChunk updating self version to: 2|1||5499a9f100df492984190492 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:44:21.589+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:21-5499a9f59a8f659f34bfba01", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356661589), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:44:21.660+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:21.664+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:44:21.665+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 }, with opId: 25465913 2014-12-23T17:44:21.665+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: -3074457345618258602 } -> { shardkey: 3074457345618258602 } 2014-12-23T17:44:21.665+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:21.705+0000 I SHARDING [conn12] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: true, waitForDelete: false, maxTimeMS: 0, epoch: ObjectId('5499a9f100df492984190492') } 2014-12-23T17:44:21.707+0000 W SHARDING [conn12] could not acquire collection lock for benchdb1.COL-0 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in benchdb1.COL-0 is taken. 2014-12-23T17:44:21.707+0000 I SHARDING [conn12] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:21-5499a9f59a8f659f34bfba02", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:42583", time: new Date(1419356661707), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, step 1 of 6: 0, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "could not acquire collection lock for benchdb1.COL-0 to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey..." } } 2014-12-23T17:44:21.938+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:21.938+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:21-5499a9f59a8f659f34bfba03", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356661938), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 679, step 3 of 6: 1, step 4 of 6: 8, step 5 of 6: 393, step 6 of 6: 1, to: "shard0001", from: "shard0002", note: "success" } } 2014-12-23T17:44:22.002+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f100df492984190492') } ntoreturn:1 keyUpdates:0 reslen:37 1422ms 2014-12-23T17:44:22.003+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:44:22.004+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f100df492984190492') } 2014-12-23T17:44:22.611+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9f69a8f659f34bfba04 2014-12-23T17:44:22.611+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:22-5499a9f69a8f659f34bfba05", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356662611), what: "moveChunk.start", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:44:22.673+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 2|0||5499a9f100df492984190492, current metadata version is 2|0||5499a9f100df492984190492 2014-12-23T17:44:22.674+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 2|0||5499a9f100df492984190492 to shard version 2|1||5499a9f100df492984190492 2014-12-23T17:44:22.674+0000 I SHARDING [conn1] collection version was loaded at version 2|1||5499a9f100df492984190492, took 0ms 2014-12-23T17:44:22.674+0000 I SHARDING [conn1] moveChunk request accepted at version 2|1||5499a9f100df492984190492 2014-12-23T17:44:22.684+0000 I SHARDING [conn1] moveChunk number of documents: 0 2014-12-23T17:44:22.687+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:22.689+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:22.694+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:22.702+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:22.719+0000 I SHARDING [conn1] moveChunk data transfer progress: { active: true, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 2014-12-23T17:44:22.719+0000 I SHARDING [conn1] About to check if it is safe to enter critical section 2014-12-23T17:44:22.719+0000 I SHARDING [conn1] About to enter migrate critical section 2014-12-23T17:44:22.719+0000 I SHARDING [conn1] moveChunk setting version to: 3|0||5499a9f100df492984190492 2014-12-23T17:44:22.780+0000 I SHARDING [conn1] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, shardKeyPattern: { shardkey: "hashed" }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } 2014-12-23T17:44:22.780+0000 I SHARDING [conn1] moveChunk updating self version to: 3|1||5499a9f100df492984190492 through { shardkey: MinKey } -> { shardkey: -3074457345618258602 } for collection 'benchdb1.COL-0' 2014-12-23T17:44:23.086+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:23-5499a9f79a8f659f34bfba06", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356663086), what: "moveChunk.commit", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000", cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 } } 2014-12-23T17:44:23.152+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:23.162+0000 I SHARDING [conn1] doing delete inline for cleanup of chunk data 2014-12-23T17:44:23.162+0000 I SHARDING [conn1] Deleter starting delete for: benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey }, with opId: 25557403 2014-12-23T17:44:23.162+0000 I SHARDING [conn1] rangeDeleter deleted 0 documents for benchdb1.COL-0 from { shardkey: 3074457345618258602 } -> { shardkey: MaxKey } 2014-12-23T17:44:23.162+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:23.419+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:23.419+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:23-5499a9f79a8f659f34bfba07", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356663419), what: "moveChunk.from", ns: "benchdb1.COL-0", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 670, step 3 of 6: 10, step 4 of 6: 33, step 5 of 6: 443, step 6 of 6: 0, to: "shard0000", from: "shard0002", note: "success" } } 2014-12-23T17:44:23.526+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-0", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-0-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f100df492984190492') } ntoreturn:1 keyUpdates:0 reslen:37 1522ms 2014-12-23T17:44:23.528+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9f100df492984190492') } 2014-12-23T17:44:23.679+0000 I SHARDING [LockPinger] cluster ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019 pinged successfully at Tue Dec 23 17:44:23 2014 by distributed lock pinger 'ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27019/ip-10-238-44-130:27018:1419353520:1363056040', sleeping for 30000ms 2014-12-23T17:44:24.133+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9f79a8f659f34bfba08 2014-12-23T17:44:24.133+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 based on current shard version 3|0||5499a9f100df492984190492, current metadata version is 3|0||5499a9f100df492984190492 2014-12-23T17:44:24.134+0000 I SHARDING [conn1] updating metadata for benchdb1.COL-0 from shard version 3|0||5499a9f100df492984190492 to shard version 3|1||5499a9f100df492984190492 2014-12-23T17:44:24.134+0000 I SHARDING [conn1] collection version was loaded at version 3|1||5499a9f100df492984190492, took 0ms 2014-12-23T17:44:24.134+0000 I SHARDING [conn1] splitChunk accepted at version 3|1||5499a9f100df492984190492 2014-12-23T17:44:24.439+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:24-5499a9f89a8f659f34bfba09", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356664439), what: "split", ns: "benchdb1.COL-0", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('5499a9f100df492984190492') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('5499a9f100df492984190492') } } } 2014-12-23T17:44:24.844+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-0/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:24.844+0000 I QUERY [conn1] command admin.$cmd command: splitChunk { splitChunk: "benchdb1.COL-0", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-0-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9f100df492984190492') } ntoreturn:1 keyUpdates:0 reslen:37 1316ms 2014-12-23T17:44:26.501+0000 I INDEX [conn9] build index on: benchdb1.COL-1 properties: { v: 1, key: { shardkey: "hashed" }, name: "shardkey_hashed", ns: "benchdb1.COL-1" } 2014-12-23T17:44:26.501+0000 I INDEX [conn9] building index using bulk method 2014-12-23T17:44:26.502+0000 I INDEX [conn9] build index done. scanned 0 total records. 0 secs 2014-12-23T17:44:26.621+0000 I COMMAND [conn1] CMD: drop benchdb1.COL-0 2014-12-23T17:44:26.727+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-0 with requested shard version 0|0||000000000000000000000000, current shard version is 3|3||5499a9f100df492984190492, current metadata version is 3|3||5499a9f100df492984190492 2014-12-23T17:44:26.728+0000 W SHARDING [conn1] no chunks found when reloading benchdb1.COL-0, previous version was 0|0||5499a9f100df492984190492, this is a drop 2014-12-23T17:44:26.728+0000 I SHARDING [conn1] dropping metadata for benchdb1.COL-0 at shard version 3|3||5499a9f100df492984190492, took 0ms 2014-12-23T17:44:26.897+0000 I SHARDING [conn9] remotely refreshing metadata for benchdb1.COL-1 with requested shard version 1|2||5499a9f700df49298419049a, current shard version is 0|0||000000000000000000000000, current metadata version is 0|0||000000000000000000000000 2014-12-23T17:44:26.898+0000 I SHARDING [conn9] collection benchdb1.COL-1 was previously unsharded, new metadata loaded with shard version 1|2||5499a9f700df49298419049a 2014-12-23T17:44:26.898+0000 I SHARDING [conn9] collection version was loaded at version 1|2||5499a9f700df49298419049a, took 0ms 2014-12-23T17:44:26.942+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:44:26.943+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f700df49298419049a') } 2014-12-23T17:44:27.105+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:44:27.135+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:44:27.167+0000 I COMMAND [conn6] dropDatabase benchdb1 starting 2014-12-23T17:44:27.168+0000 I COMMAND [conn6] dropDatabase benchdb1 finished 2014-12-23T17:44:27.486+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9fa9a8f659f34bfba0b 2014-12-23T17:44:27.486+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:27-5499a9fb9a8f659f34bfba0c", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356667486), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, from: "shard0002", to: "shard0001" } } 2014-12-23T17:44:27.548+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||5499a9f700df49298419049a, current metadata version is 1|2||5499a9f700df49298419049a 2014-12-23T17:44:27.549+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||5499a9f700df49298419049a, took 0ms) 2014-12-23T17:44:27.549+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a9f700df49298419049a 2014-12-23T17:44:27.552+0000 W SHARDING [conn1] ns not found, should be impossible 2014-12-23T17:44:27.552+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:27.855+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:27.856+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:27-5499a9fb9a8f659f34bfba0d", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356667855), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, step 1 of 6: 0, step 2 of 6: 606, to: "shard0001", from: "shard0002", note: "aborted", errmsg: "ns not found, should be impossible" } } 2014-12-23T17:44:27.914+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0001", min: { shardkey: -3074457345618258602 }, max: { shardkey: 3074457345618258602 }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_-3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f700df49298419049a') } ntoreturn:1 keyUpdates:0 reslen:84 972ms 2014-12-23T17:44:27.915+0000 I SHARDING [conn1] moveChunk waiting for full cleanup after move 2014-12-23T17:44:27.916+0000 I SHARDING [conn1] received moveChunk request: { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f700df49298419049a') } 2014-12-23T17:44:28.523+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9fb9a8f659f34bfba0e 2014-12-23T17:44:28.524+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:28-5499a9fc9a8f659f34bfba0f", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356668524), what: "moveChunk.start", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, from: "shard0002", to: "shard0000" } } 2014-12-23T17:44:28.592+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||5499a9f700df49298419049a, current metadata version is 1|2||5499a9f700df49298419049a 2014-12-23T17:44:28.593+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||5499a9f700df49298419049a, took 0ms) 2014-12-23T17:44:28.593+0000 I SHARDING [conn1] moveChunk request accepted at version 1|2||5499a9f700df49298419049a 2014-12-23T17:44:28.603+0000 W SHARDING [conn1] ns not found, should be impossible 2014-12-23T17:44:28.603+0000 I SHARDING [conn1] MigrateFromStatus::done About to acquire global lock to exit critical section 2014-12-23T17:44:28.889+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' unlocked. 2014-12-23T17:44:28.889+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:28-5499a9fc9a8f659f34bfba10", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356668889), what: "moveChunk.from", ns: "benchdb1.COL-1", details: { min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, step 1 of 6: 0, step 2 of 6: 677, to: "shard0000", from: "shard0002", note: "aborted", errmsg: "ns not found, should be impossible" } } 2014-12-23T17:44:28.947+0000 I QUERY [conn1] command admin.$cmd command: moveChunk { moveChunk: "benchdb1.COL-1", from: "ec2-54-184-200-213.us-west-2.compute.amazonaws.com:27018", to: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27018", fromShard: "shard0002", toShard: "shard0000", min: { shardkey: 3074457345618258602 }, max: { shardkey: MaxKey }, maxChunkSizeBytes: 67108864, shardId: "benchdb1.COL-1-shardkey_3074457345618258602", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", secondaryThrottle: false, waitForDelete: true, maxTimeMS: 0, epoch: ObjectId('5499a9f700df49298419049a') } ntoreturn:1 keyUpdates:0 reslen:84 1031ms 2014-12-23T17:44:28.949+0000 I SHARDING [conn1] received splitChunk request: { splitChunk: "benchdb1.COL-1", keyPattern: { shardkey: "hashed" }, min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 }, from: "shard0002", splitKeys: [ { shardkey: -6148914691236517204 } ], shardId: "benchdb1.COL-1-shardkey_MinKey", configdb: "ec2-54-189-15-112.us-west-2.compute.amazonaws.com:27019,ec2-54-185-251-41.us-west-2.compute.amazonaws.com:27019,ec2-54-184-200-213.us-west-2.compute.a...", epoch: ObjectId('5499a9f700df49298419049a') } 2014-12-23T17:44:29.556+0000 I SHARDING [conn1] distributed lock 'benchdb1.COL-1/ip-10-238-44-130:27018:1419353520:1363056040' acquired, ts : 5499a9fc9a8f659f34bfba11 2014-12-23T17:44:29.556+0000 I SHARDING [conn1] remotely refreshing metadata for benchdb1.COL-1 based on current shard version 1|2||5499a9f700df49298419049a, current metadata version is 1|2||5499a9f700df49298419049a 2014-12-23T17:44:29.557+0000 I SHARDING [conn1] metadata of collection benchdb1.COL-1 already up to date (shard version : 1|2||5499a9f700df49298419049a, took 0ms) 2014-12-23T17:44:29.557+0000 I SHARDING [conn1] splitChunk accepted at version 1|2||5499a9f700df49298419049a 2014-12-23T17:44:29.864+0000 I SHARDING [conn1] about to log metadata event: { _id: "ip-10-238-44-130-2014-12-23T17:44:29-5499a9fd9a8f659f34bfba12", server: "ip-10-238-44-130", clientAddr: "10.233.133.124:41419", time: new Date(1419356669864), what: "split", ns: "benchdb1.COL-1", details: { before: { min: { shardkey: MinKey }, max: { shardkey: -3074457345618258602 } }, left: { min: { shardkey: MinKey }, max: { shardkey: -6148914691236517204 }, lastmod: Timestamp 1000|3, lastmodEpoch: ObjectId('5499a9f700df49298419049a') }, right: { min: { shardkey: -6148914691236517204 }, max: { shardkey: -3074457345618258602 }, lastmod: Timestamp 1000|4, lastmodEpoch: ObjectId('5499a9f700df49298419049a') } } } 2014-12-23T17:44:29.961+0000 I - [conn1] Invariant failure collection src/mongo/s/d_split.cpp 842 2014-12-23T17:44:29.984+0000 I CONTROL [conn1] 0xf0bd99 0xeb5bb1 0xe9b312 0xdb85dd 0x9a9784 0x9aa5d3 0x9ab08b 0xb77c2a 0xa8af55 0x7e1770 0xec9d61 0x7f94a3128182 0x7f94a2228fbd ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B0BD99"},{"b":"400000","o":"AB5BB1"},{"b":"400000","o":"A9B312"},{"b":"400000","o":"9B85DD"},{"b":"400000","o":"5A9784"},{"b":"400000","o":"5AA5D3"},{"b":"400000","o":"5AB08B"},{"b":"400000","o":"777C2A"},{"b":"400000","o":"68AF55"},{"b":"400000","o":"3E1770"},{"b":"400000","o":"AC9D61"},{"b":"7F94A3120000","o":"8182"},{"b":"7F94A212E000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc3", "gitVersion" : "2d679247f17dab05a492c8b6d2c250dab18e54f2", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFDE3EA000", "elfType" : 3 }, { "b" : "7F94A3120000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F94A2F18000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F94A2D14000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F94A2A10000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F94A270A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F94A24F4000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F94A212E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F94A333E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf0bd99] mongod(_ZN5mongo10logContextEPKc+0xE1) [0xeb5bb1] mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xe9b312] mongod(_ZN5mongo17SplitChunkCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x311D) [0xdb85dd] mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9a9784] mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC13) [0x9aa5d3] mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9ab08b] mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERNS_5CurOpES3_b+0x76A) [0xb77c2a] mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xB25) [0xa8af55] mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7e1770] mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x411) [0xec9d61] libpthread.so.0(+0x8182) [0x7f94a3128182] libc.so.6(clone+0x6D) [0x7f94a2228fbd] ----- END BACKTRACE ----- 2014-12-23T17:44:29.984+0000 I - [conn1] ***aborting after invariant() failure