Fri Nov 22 10:59:32.937 [initandlisten] MongoDB starting : pid=15857 port=27001 dbpath=/shared/mongodb/data/db/dicom/dicom1 64-bit host=node40.clus.cci.emory.edu Fri Nov 22 10:59:32.940 [initandlisten] db version v2.4.1 Fri Nov 22 10:59:32.940 [initandlisten] git version: 1560959e9ce11a693be8b4d0d160d633eee75110 Fri Nov 22 10:59:32.940 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Fri Nov 22 10:59:32.940 [initandlisten] allocator: tcmalloc Fri Nov 22 10:59:32.940 [initandlisten] options: { dbpath: "/shared/mongodb/data/db/dicom/dicom1", directoryperdb: true, fork: true, logappend: true, logpath: "/shared/mongodb/data/log/dicom1.log", maxConns: 10000, oplogSize: 1000, port: 27001, quiet: true, replSet: "dicom1", shardsvr: true } Fri Nov 22 10:59:32.983 [initandlisten] journal dir=/shared/mongodb/data/db/dicom/dicom1/journal Fri Nov 22 10:59:32.983 [initandlisten] recover : no journal files present, no recovery needed Fri Nov 22 10:59:33.342 [initandlisten] preallocateIsFaster=true 6.02 Fri Nov 22 10:59:33.493 [initandlisten] preallocateIsFaster=true 2.06 Fri Nov 22 10:59:34.694 [initandlisten] preallocateIsFaster=true 2.42 Fri Nov 22 10:59:34.694 [initandlisten] preallocating a journal file /shared/mongodb/data/db/dicom/dicom1/journal/prealloc.0 Fri Nov 22 10:59:46.823 [initandlisten] preallocating a journal file /shared/mongodb/data/db/dicom/dicom1/journal/prealloc.1 Fri Nov 22 10:59:58.738 [initandlisten] preallocating a journal file /shared/mongodb/data/db/dicom/dicom1/journal/prealloc.2 Fri Nov 22 11:00:01.012 [initandlisten] File Preallocator Progress: 985661440/1073741824 91% Fri Nov 22 11:00:09.533 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/local/local.ns, filling with zeroes... Fri Nov 22 11:00:09.533 [FileAllocator] creating directory /shared/mongodb/data/db/dicom/dicom1/local/_tmp Fri Nov 22 11:00:09.897 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/local/local.ns, size: 16MB, took 0.334 secs Fri Nov 22 11:00:09.897 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/local/local.0, filling with zeroes... Fri Nov 22 11:00:10.675 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/local/local.0, size: 64MB, took 0.777 secs Fri Nov 22 11:00:10.679 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 1147ms Fri Nov 22 11:00:10.680 [initandlisten] waiting for connections on port 27001 Fri Nov 22 11:00:10.680 [websvr] admin web console waiting for connections on port 28001 Fri Nov 22 11:00:10.688 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:00:10.688 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done Fri Nov 22 11:00:20.690 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:00:30.693 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:00:40.695 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:00:50.695 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:01:00.696 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:01:10.699 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:01:20.701 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:01:30.701 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:01:40.703 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:01:50.706 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:02:00.730 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:02:10.732 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:02:20.734 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:02:30.735 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:02:40.736 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:02:50.738 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:03:00.740 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:03:10.740 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:03:20.742 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:03:30.744 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:03:40.745 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:03:50.746 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:04:00.748 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:04:10.750 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:04:20.751 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:04:30.752 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:04:40.849 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:04:50.851 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:05:00.852 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:05:10.854 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:05:20.901 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:05:30.902 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:05:40.903 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:05:50.905 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:06:00.907 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:06:10.908 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:06:20.909 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:06:30.910 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:06:40.911 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:06:50.912 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:07:00.913 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:07:10.915 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:07:20.917 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:07:30.918 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:07:40.920 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:07:50.922 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:08:00.923 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:08:10.924 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:08:20.926 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:08:30.928 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:08:40.929 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:08:50.931 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:09:01.023 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:09:11.025 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:09:21.026 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:09:27.059 got signal 15 (Terminated), will terminate after current cmd ends Fri Nov 22 11:09:28.234 [interruptThread] now exiting Fri Nov 22 11:09:28.234 dbexit: Fri Nov 22 11:09:28.234 [interruptThread] shutdown: going to close listening sockets... Fri Nov 22 11:09:28.234 [interruptThread] closing listening socket: 9 Fri Nov 22 11:09:28.234 [interruptThread] closing listening socket: 10 Fri Nov 22 11:09:28.234 [interruptThread] closing listening socket: 11 Fri Nov 22 11:09:28.234 [interruptThread] removing socket file: /tmp/mongodb-27001.sock Fri Nov 22 11:09:28.234 [interruptThread] shutdown: going to flush diaglog... Fri Nov 22 11:09:28.234 [interruptThread] shutdown: going to close sockets... Fri Nov 22 11:09:28.234 [interruptThread] shutdown: waiting for fs preallocator... Fri Nov 22 11:09:28.234 [interruptThread] shutdown: lock for final commit... Fri Nov 22 11:09:28.234 [interruptThread] shutdown: final commit... Fri Nov 22 11:09:28.234 [interruptThread] shutdown: closing all files... Fri Nov 22 11:09:28.235 [interruptThread] closeAllFiles() finished Fri Nov 22 11:09:28.235 [interruptThread] journalCleanup... Fri Nov 22 11:09:28.235 [interruptThread] removeJournalFiles Fri Nov 22 11:09:32.926 [DataFileSync] ERROR: Client::shutdown not called: DataFileSync Fri Nov 22 11:09:34.344 [interruptThread] shutdown: removing fs lock... Fri Nov 22 11:09:34.344 dbexit: really exiting now ***** SERVER RESTARTED ***** Fri Nov 22 11:10:47.188 [initandlisten] MongoDB starting : pid=16665 port=27001 dbpath=/shared/mongodb/data/db/dicom/dicom1 64-bit host=node40.clus.cci.emory.edu Fri Nov 22 11:10:47.192 [initandlisten] db version v2.4.1 Fri Nov 22 11:10:47.192 [initandlisten] git version: 1560959e9ce11a693be8b4d0d160d633eee75110 Fri Nov 22 11:10:47.192 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Fri Nov 22 11:10:47.192 [initandlisten] allocator: tcmalloc Fri Nov 22 11:10:47.192 [initandlisten] options: { dbpath: "/shared/mongodb/data/db/dicom/dicom1", directoryperdb: true, fork: true, logappend: true, logpath: "/shared/mongodb/data/log/dicom1.log", maxConns: 10000, oplogSize: 1000, port: 27001, quiet: true, replSet: "dicom1", shardsvr: true } Fri Nov 22 11:10:47.225 [initandlisten] journal dir=/shared/mongodb/data/db/dicom/dicom1/journal Fri Nov 22 11:10:47.226 [initandlisten] recover : no journal files present, no recovery needed Fri Nov 22 11:10:47.313 [initandlisten] waiting for connections on port 27001 Fri Nov 22 11:10:47.313 [websvr] admin web console waiting for connections on port 28001 Fri Nov 22 11:10:47.320 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:10:47.320 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done Fri Nov 22 11:10:57.323 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:11:07.324 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:11:17.325 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:11:27.327 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:11:37.329 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:11:47.330 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:11:57.331 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:12:07.333 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:12:17.335 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:12:27.336 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:12:37.337 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:12:47.338 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:12:57.339 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:13:07.340 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:13:17.341 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:13:27.342 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:13:37.343 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:13:47.344 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:13:57.344 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:14:07.347 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:14:17.348 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:14:27.348 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:14:37.350 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:14:47.352 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:14:57.352 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:15:07.353 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:15:17.356 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:15:27.356 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:15:37.357 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:15:47.358 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:15:57.360 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:16:07.360 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:16:17.361 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:16:27.363 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Fri Nov 22 11:16:34.266 [conn1] replSet replSetInitiate admin command received from client Fri Nov 22 11:16:34.292 [conn1] replSet info initiate : no configuration specified. Using a default configuration for the set Fri Nov 22 11:16:34.292 [conn1] replSet created this configuration for initiation : { _id: "dicom1", members: [ { _id: 0, host: "node40.clus.cci.emory.edu:27001" } ] } Fri Nov 22 11:16:34.292 [conn1] replSet replSetInitiate config object parses ok, 1 members specified Fri Nov 22 11:16:34.294 [conn1] replSet replSetInitiate all members seem up Fri Nov 22 11:16:34.294 [conn1] ****** Fri Nov 22 11:16:34.294 [conn1] creating replication oplog of size: 1000MB... Fri Nov 22 11:16:34.295 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/local/local.1, filling with zeroes... Fri Nov 22 11:16:46.310 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/local/local.1, size: 1024MB, took 12.014 secs Fri Nov 22 11:16:46.942 [conn1] ****** Fri Nov 22 11:16:46.942 [conn1] replSet info saving a newer config version to local.system.replset Fri Nov 22 11:16:46.945 [conn1] replSet saveConfigLocally done Fri Nov 22 11:16:46.945 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. Fri Nov 22 11:16:46.945 [conn1] command admin.$cmd command: { replSetInitiate: undefined } ntoreturn:1 keyUpdates:0 locks(micros) W:12676965 reslen:215 12679ms Fri Nov 22 11:16:46.946 [rsStart] replSet I am node40.clus.cci.emory.edu:27001 Fri Nov 22 11:16:46.946 [rsStart] replSet STARTUP2 Fri Nov 22 11:16:47.962 [rsSync] replSet SECONDARY Fri Nov 22 11:16:47.962 [rsMgr] replSet info electSelf 0 Fri Nov 22 11:16:48.962 [rsMgr] replSet PRIMARY Fri Nov 22 11:35:13.884 [conn8] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers Fri Nov 22 11:40:13.923 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 11:45:13.963 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 11:50:14.002 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 11:55:14.041 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:00:14.080 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:05:14.119 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:10:14.158 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:12:10.537 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.ns, filling with zeroes... Fri Nov 22 12:12:10.538 [FileAllocator] creating directory /shared/mongodb/data/db/dicom/dicom1/dicomdb/_tmp Fri Nov 22 12:12:10.825 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.ns, size: 16MB, took 0.258 secs Fri Nov 22 12:12:10.825 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.0, filling with zeroes... Fri Nov 22 12:12:11.679 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.0, size: 64MB, took 0.853 secs Fri Nov 22 12:12:11.680 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.1, filling with zeroes... Fri Nov 22 12:12:11.684 [conn7] build index dicomdb.fs.chunks { _id: 1 } Fri Nov 22 12:12:11.685 [conn7] build index done. scanned 0 total records. 0.001 secs Fri Nov 22 12:12:11.686 [conn7] info: creating collection dicomdb.fs.chunks on add index Fri Nov 22 12:12:11.686 [conn7] build index dicomdb.fs.chunks { files_id: 1.0 } Fri Nov 22 12:12:11.687 [conn7] build index done. scanned 0 total records. 0.001 secs Fri Nov 22 12:12:11.687 [conn7] insert dicomdb.system.indexes ninserted:1 keyUpdates:0 locks(micros) w:1150397 1150ms Fri Nov 22 12:12:13.079 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.1, size: 128MB, took 1.399 secs Fri Nov 22 12:12:13.832 [conn11] no current chunk manager found for this shard, will initialize Fri Nov 22 12:12:13.832 [conn11] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Fri Nov 22 12:12:13.833 [conn11] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Fri Nov 22 12:12:13.834 [conn11] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Fri Nov 22 12:12:13.914 [conn7] build index dicomdb.dicom { _id: 1 } Fri Nov 22 12:12:13.915 [conn7] build index done. scanned 0 total records. 0.001 secs Fri Nov 22 12:12:13.916 [conn7] info: creating collection dicomdb.dicom on add index Fri Nov 22 12:12:13.916 [conn7] build index dicomdb.dicom { files_id: 1.0 } Fri Nov 22 12:12:13.917 [conn7] build index done. scanned 0 total records. 0.001 secs Fri Nov 22 12:12:14.520 [conn11] no current chunk manager found for this shard, will initialize Fri Nov 22 12:12:14.561 [conn7] build index dicomdb.fs.files { _id: 1 } Fri Nov 22 12:12:14.562 [conn7] build index done. scanned 0 total records. 0.001 secs Fri Nov 22 12:12:14.562 [conn7] info: creating collection dicomdb.fs.files on add index Fri Nov 22 12:12:14.838 [conn11] no current chunk manager found for this shard, will initialize Fri Nov 22 12:15:14.197 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:20:14.236 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:25:14.275 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:30:14.314 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:35:14.352 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:40:14.392 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:45:14.431 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 12:47:14.761 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 12:50:14.470 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:52:14.801 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:55:14.509 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 12:57:14.840 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:00:14.548 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:02:14.878 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:05:14.586 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 13:07:14.917 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:10:14.626 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:12:14.957 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:15:14.665 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:17:14.996 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:20:14.703 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:22:15.035 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:25:14.742 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:27:15.074 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:30:14.781 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:32:15.112 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:35:14.819 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:37:15.151 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:40:14.859 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:42:15.200 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:45:14.898 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:47:15.239 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 13:50:14.936 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:52:15.278 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:55:14.975 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 13:57:15.318 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 14:00:15.014 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:02:15.358 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 14:05:15.053 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:07:15.396 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:10:15.092 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:12:15.435 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:15:15.131 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:17:15.474 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:20:15.170 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:22:15.513 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:25:15.209 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:27:15.552 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:30:15.247 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:32:15.591 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:35:15.286 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 14:37:15.630 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:40:15.325 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:42:15.669 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:45:15.364 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:47:15.708 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:50:15.402 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:52:15.747 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:55:15.441 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 14:57:15.786 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 15:00:15.479 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 15:02:15.825 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:05:15.518 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:07:15.864 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:10:15.557 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:12:15.903 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:15:15.596 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:17:15.942 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:20:15.635 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:22:15.981 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:25:15.673 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:27:16.021 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:30:15.713 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:32:16.059 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:35:15.751 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:37:16.099 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:40:15.791 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 15:42:16.138 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:45:15.830 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:47:16.177 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 15:50:15.868 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:52:16.216 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:55:15.907 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 15:57:16.255 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:00:15.946 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:02:16.294 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:05:15.985 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:07:16.332 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 16:10:16.024 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:12:16.371 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:15:16.064 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 16:17:16.410 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:20:16.102 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:22:16.450 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 16:25:16.141 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:27:16.489 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:30:16.180 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:32:16.527 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:35:16.218 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:37:16.567 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:40:16.258 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 16:42:16.606 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:45:16.296 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:47:16.645 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:50:16.335 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 16:52:16.683 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:55:16.373 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 16:57:16.722 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:00:16.412 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:02:16.761 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:05:16.451 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:07:16.799 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:10:16.490 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:12:16.838 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:15:16.529 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:17:16.876 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:20:16.567 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:22:16.915 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:25:16.606 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:27:16.954 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:30:16.645 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:32:16.992 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:35:16.683 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:37:17.031 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:40:16.721 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:42:17.069 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:45:16.759 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:47:17.108 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:50:16.797 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 17:52:17.147 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:55:16.837 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 17:57:17.186 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:00:16.875 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:02:17.225 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:05:16.914 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:07:17.264 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:10:16.953 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:12:17.304 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:15:16.992 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:17:17.343 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:20:17.031 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:22:17.382 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:25:17.070 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:27:17.422 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:30:17.110 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:32:17.462 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:35:17.149 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:37:17.501 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:40:17.189 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:42:17.540 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:45:17.228 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:47:17.580 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 18:50:17.266 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:52:17.619 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:55:17.305 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 18:57:17.658 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:00:17.364 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:02:17.697 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:05:17.403 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 19:07:17.736 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:10:17.442 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:12:17.775 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:15:17.481 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:17:17.815 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 19:20:17.520 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 19:22:17.853 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:25:17.558 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:27:17.893 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:30:17.598 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:32:17.932 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:35:17.637 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:37:17.971 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:40:17.675 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:42:18.010 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:45:17.714 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:47:18.049 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 19:50:17.753 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:52:18.089 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:55:17.792 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 19:57:18.127 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:00:17.831 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:02:18.166 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:05:17.871 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 20:07:18.206 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:10:17.910 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:12:18.245 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 20:15:17.948 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:17:18.284 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:20:17.988 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:22:18.323 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:25:18.026 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 20:27:18.362 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:30:18.065 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 20:32:18.401 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:35:18.105 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:37:18.441 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 20:40:18.143 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:42:18.480 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:45:18.182 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:47:18.519 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:50:18.220 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 20:52:18.558 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:55:18.259 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 20:57:18.597 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 21:00:18.299 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:02:18.636 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:05:18.338 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:07:18.675 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:10:18.377 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:12:18.715 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:15:18.417 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:17:18.755 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:20:18.457 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:22:18.793 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:25:18.496 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:27:18.833 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:30:18.536 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:32:18.872 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:35:18.576 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:37:18.912 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 21:40:18.614 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:42:18.951 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:45:18.653 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:47:18.990 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:50:18.693 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:52:19.029 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:55:18.732 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 21:57:19.069 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 22:00:18.771 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:02:19.108 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 22:05:18.809 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:07:19.148 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 22:10:18.848 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:12:19.187 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:15:18.887 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:17:19.225 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:20:18.926 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:22:19.264 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:25:18.966 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:27:19.304 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:30:19.004 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:32:19.342 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:35:19.043 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:37:19.381 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:40:19.082 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:42:19.419 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 22:45:19.120 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 22:47:19.458 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:50:19.159 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:52:19.496 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Fri Nov 22 22:55:19.198 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 22:57:19.535 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:00:19.237 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:02:19.574 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:05:19.275 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:07:19.614 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:10:19.314 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:11:52.092 [conn27] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:11:52.280 [conn42] recv(): message len 1414876743 is too large. Max is 48000000 Fri Nov 22 23:11:52.282 [conn44] recv(): message len 1414876743 is too large. Max is 48000000 Fri Nov 22 23:11:52.283 [conn45] recv(): message len 1347175752 is too large. Max is 48000000 Fri Nov 22 23:11:52.284 [conn46] recv(): message len 1145128264 is too large. Max is 48000000 Fri Nov 22 23:11:52.286 [conn47] recv(): message len 1145128264 is too large. Max is 48000000 Fri Nov 22 23:11:52.287 [conn48] recv(): message len 1145128264 is too large. Max is 48000000 Fri Nov 22 23:11:52.467 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:50189 Fri Nov 22 23:11:52.511 [conn86] recv(): message len 1969448275 is too large. Max is 48000000 Fri Nov 22 23:11:52.535 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:50262 Fri Nov 22 23:11:52.643 [conn120] recv(): message len 1331317580 is too large. Max is 48000000 Fri Nov 22 23:11:52.735 [conn124] recv(): message len 1701670771 is too large. Max is 48000000 Fri Nov 22 23:12:02.887 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:57675 Fri Nov 22 23:12:03.032 [conn182] recv(): message len 1230262351 is too large. Max is 48000000 Fri Nov 22 23:12:04.530 [conn233] recv(): message len 1313754947 is too large. Max is 48000000 Fri Nov 22 23:12:04.552 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:36522 Fri Nov 22 23:12:05.102 [conn257] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:12:07.624 [conn318] recv(): message len 1230262351 is too large. Max is 48000000 Fri Nov 22 23:12:07.803 [conn327] recv(): message len 1347375696 is too large. Max is 48000000 Fri Nov 22 23:12:07.814 [conn329] recv(): message len 1145128264 is too large. Max is 48000000 Fri Nov 22 23:12:07.816 [conn330] recv(): message len 1145128264 is too large. Max is 48000000 Fri Nov 22 23:12:08.956 [conn417] recv(): message len 1230262351 is too large. Max is 48000000 Fri Nov 22 23:12:14.291 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:45325 Fri Nov 22 23:12:14.298 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:45347 Fri Nov 22 23:12:14.304 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:45397 Fri Nov 22 23:12:19.653 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:12:20.095 [conn702] recv(): message len 1128354388 is too large. Max is 48000000 Fri Nov 22 23:12:20.097 [conn703] recv(): message len 1128354388 is too large. Max is 48000000 Fri Nov 22 23:12:20.249 [conn716] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:12:26.416 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:54981 Fri Nov 22 23:12:26.416 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:55026 Fri Nov 22 23:12:26.423 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:55045 Fri Nov 22 23:12:26.429 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:55081 Fri Nov 22 23:12:33.187 [conn810] recv(): message len 1397966158 is too large. Max is 48000000 Fri Nov 22 23:12:35.025 [conn989] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:12:37.470 [conn1228] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:12:38.451 [conn1261] recv(): message len 1380009299 is too large. Max is 48000000 Fri Nov 22 23:12:45.777 [conn1502] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:12:56.263 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:51055 Fri Nov 22 23:12:56.263 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:51210 Fri Nov 22 23:12:56.264 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:51221 Fri Nov 22 23:13:01.871 [conn1619] recv(): message len 1230262351 is too large. Max is 48000000 Fri Nov 22 23:13:01.931 [conn1621] recv(): message len 542397776 is too large. Max is 48000000 Fri Nov 22 23:13:02.789 [conn1640] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:13:03.450 [conn1661] recv(): message len 1414745936 is too large. Max is 48000000 Fri Nov 22 23:13:22.147 [websvr] Socket http response send() errno:11 Resource temporarily unavailable 10.224.10.20:56909 Fri Nov 22 23:13:22.154 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:56915 Fri Nov 22 23:13:22.154 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:57625 Fri Nov 22 23:13:22.160 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:57801 Fri Nov 22 23:13:22.161 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:40086 Fri Nov 22 23:13:22.161 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:43774 Fri Nov 22 23:13:22.167 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:43838 Fri Nov 22 23:13:22.173 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:43855 Fri Nov 22 23:15:19.353 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:17:19.691 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:20:19.392 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:22:19.730 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:25:19.431 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:27:19.769 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:30:19.470 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:32:19.808 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:35:19.509 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:37:19.847 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:40:19.547 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:42:19.886 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:45:19.587 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:47:19.925 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:50:19.626 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:52:19.964 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Fri Nov 22 23:55:19.665 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Fri Nov 22 23:57:20.003 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:00:19.704 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:02:20.043 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 00:05:19.743 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:07:20.082 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:10:19.782 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:12:20.120 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:15:19.820 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 00:17:20.159 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:20:19.860 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:22:20.199 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:25:19.899 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:27:20.238 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:30:19.938 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 00:32:20.277 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:35:19.977 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:37:20.317 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 00:40:20.016 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:42:20.356 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:45:20.055 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:47:20.394 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:50:20.094 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:52:20.433 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 00:55:20.134 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 00:57:20.472 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:00:20.173 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:02:20.511 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:05:20.211 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:07:20.550 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:10:20.250 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:12:20.589 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:15:20.289 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:17:20.628 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:20:20.328 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:22:20.667 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:25:20.366 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:27:20.706 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:30:20.405 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:32:20.746 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 01:35:20.444 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:37:20.784 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:40:20.483 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:42:20.823 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:45:20.521 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 01:47:20.862 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:50:20.560 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:52:20.901 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:55:20.599 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 01:57:20.940 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:00:20.637 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:02:20.980 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 02:05:20.676 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:07:21.019 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:10:20.715 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:12:21.057 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:15:20.754 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 02:17:21.097 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:20:20.793 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 02:22:21.135 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 02:25:20.832 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:27:21.173 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:30:20.870 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:32:21.212 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:35:20.909 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:37:21.251 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:40:20.948 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:42:21.289 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 02:45:20.987 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:47:21.328 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:50:21.026 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:52:21.367 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:55:21.064 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 02:57:21.406 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:00:21.103 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:02:21.445 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:05:21.142 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:07:21.483 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:10:21.181 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:12:21.522 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 03:15:21.220 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:17:21.561 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:20:21.259 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 03:22:21.600 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:25:21.307 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 03:27:21.639 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:30:21.346 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:32:21.677 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:35:21.384 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:37:21.716 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:40:21.424 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:42:21.755 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:45:21.463 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:47:21.794 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:50:21.502 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 03:52:21.833 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:55:21.541 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 03:57:21.872 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:00:21.580 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:02:21.911 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:05:21.619 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:07:21.949 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 04:10:21.658 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:12:21.988 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:15:21.698 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 04:17:22.028 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:20:21.737 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:22:22.067 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 04:25:21.776 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:27:22.106 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:30:21.815 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:32:22.145 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:35:21.853 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:37:22.184 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:40:21.892 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:42:22.223 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 04:45:21.932 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:47:22.262 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:50:21.971 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:52:22.302 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 04:55:22.010 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 04:57:22.341 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:00:22.049 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:02:22.379 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:05:22.088 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:07:22.419 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:10:22.127 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 05:12:22.457 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 05:15:22.166 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:17:22.496 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:20:22.205 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:22:22.535 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:25:22.243 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:27:22.574 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:30:22.282 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:32:22.613 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:35:22.321 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 05:37:22.652 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:40:22.360 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:42:22.690 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 05:45:22.399 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:47:22.729 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:50:22.438 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 05:52:22.768 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:55:22.477 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 05:57:22.807 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:00:22.516 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:02:22.845 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:05:22.555 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:07:22.884 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:10:22.594 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:12:22.923 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:15:22.632 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 06:17:22.962 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:20:22.671 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:22:23.001 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:25:22.710 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:27:23.040 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:30:22.749 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:32:23.079 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:35:22.788 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:37:23.118 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:40:22.827 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:42:23.157 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:45:22.866 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:47:23.196 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:50:22.904 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 06:52:23.236 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 06:55:22.943 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 06:57:23.274 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:00:22.982 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:02:23.313 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:05:23.021 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:07:23.352 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:10:23.060 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:12:23.391 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:15:23.099 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:17:23.431 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 07:20:23.138 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:22:23.469 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:25:23.176 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:27:23.508 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:30:23.216 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:32:23.547 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:35:23.255 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:37:23.586 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:40:23.294 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 07:42:23.625 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:45:23.333 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:47:23.664 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:50:23.372 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:52:23.703 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:55:23.410 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 07:57:23.742 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:00:23.450 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:02:23.781 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:05:23.489 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:07:23.820 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:10:23.528 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 08:12:23.859 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:15:23.567 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:17:23.899 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 08:20:23.606 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:22:23.938 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:25:23.645 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:27:23.976 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:30:23.683 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 08:32:24.015 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:35:23.722 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:37:24.054 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:40:23.762 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:42:24.111 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:45:23.800 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:47:24.150 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:50:23.839 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:52:24.189 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:55:23.878 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 08:57:24.228 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 09:00:23.917 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:02:24.267 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:05:23.956 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:07:24.306 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:10:23.995 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:12:24.345 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:15:24.034 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:17:24.384 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:20:24.073 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:22:24.422 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:25:24.111 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 09:27:24.461 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:30:24.150 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:32:24.500 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:35:24.189 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:37:24.539 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:40:24.228 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:42:24.578 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:45:24.266 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 09:47:24.617 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:50:24.305 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 09:52:24.658 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 09:55:24.344 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 09:57:24.696 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:00:24.382 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:02:24.735 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:05:24.422 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:07:24.774 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:10:24.460 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:12:24.814 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:15:24.499 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:17:24.853 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:20:24.538 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:22:24.891 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:25:24.576 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 10:27:24.930 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:30:24.615 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:32:24.968 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 10:35:24.654 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:37:25.008 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:40:24.693 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:42:25.047 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:45:24.732 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:47:25.085 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:50:24.770 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:52:25.124 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:55:24.809 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 10:57:25.164 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 11:00:24.849 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:02:25.203 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:05:24.888 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:07:25.242 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:10:24.926 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:12:25.281 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:15:24.965 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:17:25.321 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 11:20:25.004 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:22:25.360 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:25:25.043 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:27:25.398 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:30:25.082 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:32:25.437 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:35:25.121 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:37:25.476 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 11:40:25.160 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:42:25.515 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:45:25.199 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:47:25.555 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 11:50:25.237 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:52:25.594 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:55:25.276 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 11:57:25.632 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:00:25.315 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:02:25.671 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:05:25.354 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:07:25.710 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:10:25.393 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:12:25.751 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 12:15:25.432 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:17:25.789 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:20:25.471 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:22:25.827 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 12:25:25.508 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 12:27:25.866 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:30:25.547 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:32:25.905 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:35:25.586 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:37:25.944 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:40:25.624 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:42:25.983 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 12:45:25.663 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:47:26.022 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:50:25.702 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:52:26.060 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:55:25.741 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 12:57:26.099 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:00:25.781 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 13:02:26.138 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:05:25.820 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:07:26.177 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 13:10:25.858 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:12:26.216 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:15:25.897 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:17:26.255 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:20:25.936 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:22:26.293 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:25:25.975 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:27:26.332 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:30:26.013 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:32:26.370 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:35:26.050 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:37:26.408 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:40:26.088 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:42:26.445 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:45:26.126 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:47:26.483 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:50:26.164 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 13:52:26.522 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:55:26.204 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 13:57:26.561 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:00:26.259 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:02:26.601 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:05:26.298 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:07:26.640 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:10:26.337 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:12:26.678 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:15:26.376 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:17:26.717 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:20:26.414 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 14:22:26.756 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:25:26.453 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 14:27:26.794 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:30:26.492 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:32:26.833 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:35:26.531 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:37:26.872 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 14:40:26.570 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:42:26.911 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:45:26.609 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:47:26.950 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:50:26.647 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 14:52:26.989 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 14:55:26.687 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 14:57:27.028 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:00:26.726 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:02:27.068 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 15:05:26.764 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:07:27.107 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:10:26.803 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:12:27.147 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 15:15:26.842 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:17:27.186 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:20:26.881 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 15:22:27.224 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:25:26.920 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:27:27.264 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 15:30:26.959 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:32:27.303 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:35:26.998 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:37:27.343 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 15:40:27.037 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:42:27.381 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:45:27.075 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:47:27.421 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:50:27.115 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:52:27.460 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:55:27.154 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 15:57:27.499 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 16:00:27.193 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:02:27.538 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:05:27.231 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:07:27.577 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:10:27.270 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:12:27.616 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:15:27.309 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:17:27.655 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:20:27.348 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 16:22:27.694 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:25:27.387 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 16:27:27.733 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:30:27.426 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:32:27.772 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:35:27.465 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:37:27.812 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 16:40:27.503 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:42:27.851 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:45:27.543 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:47:27.890 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:50:27.581 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 16:52:27.929 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:55:27.620 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 16:57:27.968 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:00:27.658 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:02:28.007 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:05:27.697 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:07:28.047 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 17:10:27.736 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:12:28.086 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:15:27.775 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:17:28.124 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:20:27.814 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:22:28.163 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:25:27.853 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:27:28.202 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:30:27.893 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:32:28.241 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:35:27.932 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:37:28.280 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:40:27.970 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:42:28.319 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:45:28.010 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:47:28.358 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:50:28.049 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:52:28.397 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:55:28.088 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 17:57:28.436 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 18:00:28.127 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:02:28.475 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:05:28.166 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:07:28.514 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:10:28.204 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 18:12:28.553 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:15:28.242 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:17:28.592 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:20:28.282 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 18:22:28.631 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:25:28.321 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:27:28.670 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 18:30:28.359 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:32:28.709 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:35:28.398 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:37:28.747 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 18:40:28.436 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 18:42:28.786 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:45:28.476 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:47:28.825 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 18:50:28.515 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:52:28.864 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:55:28.553 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 18:57:28.903 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:00:28.592 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:02:28.942 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:05:28.631 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:07:28.980 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:10:28.669 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 19:12:29.019 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:15:28.709 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 19:17:29.068 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:20:28.748 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:22:29.107 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 19:25:28.786 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:27:29.146 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:30:28.825 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:32:29.186 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 19:35:28.864 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:37:29.225 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:40:28.903 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:42:29.263 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:45:28.942 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:47:29.302 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:50:28.982 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 19:52:29.342 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:55:29.021 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 19:57:29.380 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:00:29.060 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:02:29.419 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:05:29.098 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:07:29.458 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:10:29.137 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:12:29.496 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 20:15:29.176 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:17:29.536 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:20:29.216 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 20:22:29.575 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:25:29.255 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:27:29.614 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:30:29.294 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:32:29.652 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:35:29.332 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 20:37:29.691 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:40:29.371 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:42:29.730 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:45:29.411 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:47:29.769 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:50:29.449 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:52:29.808 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:55:29.488 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 20:57:29.847 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:00:29.527 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:02:29.885 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:05:29.566 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:07:29.924 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:10:29.605 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:12:29.963 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:15:29.644 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 21:17:30.002 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:20:29.683 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:22:30.041 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:25:29.722 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:27:30.081 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 21:30:29.761 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:32:30.119 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:35:29.800 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:37:30.158 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 21:40:29.839 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:42:30.197 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:45:29.878 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 21:47:30.236 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:50:29.917 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:52:30.275 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:55:29.956 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 21:57:30.314 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 22:00:29.995 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:02:30.353 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:05:30.035 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 22:07:30.392 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:10:30.074 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 22:12:30.431 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:15:30.113 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:17:30.470 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:20:30.152 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:22:30.510 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 22:25:30.191 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:27:30.549 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:30:30.230 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:32:30.588 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 22:35:30.268 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 22:37:30.627 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:40:30.307 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:42:30.666 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:45:30.345 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:47:30.705 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:50:30.384 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 22:52:30.745 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 22:55:30.423 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 22:57:30.783 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:00:30.462 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:02:30.822 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:05:30.501 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:07:30.861 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sat Nov 23 23:10:30.541 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:12:30.900 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:15:30.579 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:17:30.939 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:20:30.618 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:22:30.978 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 23:25:30.656 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:27:31.017 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:30:30.695 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:32:31.057 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 23:35:30.735 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:37:31.095 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:40:30.773 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:42:31.134 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:45:30.812 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:47:31.173 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:50:30.851 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:52:31.213 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sat Nov 23 23:55:30.889 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sat Nov 23 23:57:31.252 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:00:30.928 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:02:31.290 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 00:05:30.966 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 00:07:31.329 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:09:41.201 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:33218 Sun Nov 24 00:09:57.079 [conn1802] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:10:08.075 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:47321 Sun Nov 24 00:10:08.092 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:47336 Sun Nov 24 00:10:08.099 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:47349 Sun Nov 24 00:10:08.106 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:47371 Sun Nov 24 00:10:08.150 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:47401 Sun Nov 24 00:10:08.188 [conn1826] recv(): message len 1347175752 is too large. Max is 48000000 Sun Nov 24 00:10:08.189 [conn1827] recv(): message len 1145128264 is too large. Max is 48000000 Sun Nov 24 00:10:08.193 [conn1828] recv(): message len 1145128264 is too large. Max is 48000000 Sun Nov 24 00:10:08.194 [conn1829] recv(): message len 1145128264 is too large. Max is 48000000 Sun Nov 24 00:10:08.216 [conn1834] recv(): message len 1414876743 is too large. Max is 48000000 Sun Nov 24 00:10:08.220 [conn1835] recv(): message len 1414876743 is too large. Max is 48000000 Sun Nov 24 00:10:08.369 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:47614 Sun Nov 24 00:10:13.165 [conn1884] recv(): message len 1969448275 is too large. Max is 48000000 Sun Nov 24 00:10:13.197 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:47869 Sun Nov 24 00:10:13.313 [conn1899] recv(): message len 1701670771 is too large. Max is 48000000 Sun Nov 24 00:10:13.463 [conn1915] recv(): message len 1162822213 is too large. Max is 48000000 Sun Nov 24 00:10:15.112 [conn1929] recv(): message len 1230262351 is too large. Max is 48000000 Sun Nov 24 00:10:15.318 [conn1936] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:10:16.519 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:48394 Sun Nov 24 00:10:17.470 [conn2014] recv(): message len 1230262351 is too large. Max is 48000000 Sun Nov 24 00:10:19.564 [conn2214] recv(): message len 1380009299 is too large. Max is 48000000 Sun Nov 24 00:10:20.726 [conn2331] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:10:21.679 [conn2502] recv(): message len 1230262351 is too large. Max is 48000000 Sun Nov 24 00:10:22.652 [conn2609] recv(): message len 1397966158 is too large. Max is 48000000 Sun Nov 24 00:10:22.687 [conn2614] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:10:23.072 [conn2641] recv(): message len 1145128264 is too large. Max is 48000000 Sun Nov 24 00:10:23.073 [conn2642] recv(): message len 1145128264 is too large. Max is 48000000 Sun Nov 24 00:10:24.162 [conn2793] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:10:29.518 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:53956 Sun Nov 24 00:10:29.518 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:53957 Sun Nov 24 00:10:29.534 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:54170 Sun Nov 24 00:10:31.006 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:10:35.740 [conn2873] recv(): message len 1313754947 is too large. Max is 48000000 Sun Nov 24 00:10:35.814 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:54454 Sun Nov 24 00:10:49.671 [conn3080] recv(): message len 1347375696 is too large. Max is 48000000 Sun Nov 24 00:10:49.827 [conn3092] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:10:50.181 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:56130 Sun Nov 24 00:10:50.764 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:56601 Sun Nov 24 00:10:51.210 [conn3258] recv(): message len 1128354388 is too large. Max is 48000000 Sun Nov 24 00:10:51.211 [conn3260] recv(): message len 1128354388 is too large. Max is 48000000 Sun Nov 24 00:10:55.898 [conn3443] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:10:57.258 [conn3581] recv(): message len 1230262351 is too large. Max is 48000000 Sun Nov 24 00:10:57.266 [conn3583] recv(): message len 542397776 is too large. Max is 48000000 Sun Nov 24 00:10:57.449 [conn3593] recv(): message len 1414745936 is too large. Max is 48000000 Sun Nov 24 00:11:13.822 [websvr] Socket http response send() errno:11 Resource temporarily unavailable 10.224.10.20:40356 Sun Nov 24 00:11:13.838 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:40368 Sun Nov 24 00:11:13.853 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:40371 Sun Nov 24 00:11:13.853 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:40372 Sun Nov 24 00:11:13.859 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:40488 Sun Nov 24 00:11:13.865 [websvr] Socket http response send() errno:104 Connection reset by peer 10.224.10.20:40491 Sun Nov 24 00:12:31.368 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:15:31.045 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:17:31.407 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:20:31.084 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:22:31.447 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 00:25:31.122 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 00:27:31.486 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:30:31.161 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:32:31.524 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:35:31.201 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:37:31.564 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:40:31.240 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 00:42:31.603 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:45:31.279 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:47:31.643 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 00:50:31.318 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:52:31.681 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:55:31.357 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 00:57:31.720 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:00:31.395 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:02:31.759 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:05:31.434 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:07:31.798 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:10:31.473 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:12:31.837 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:15:31.511 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:17:31.877 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 01:20:31.550 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:22:31.916 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:25:31.589 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:27:31.955 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 01:30:31.628 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:32:31.994 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:35:31.667 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:37:32.034 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:40:31.706 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:42:32.073 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:45:31.745 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:47:32.113 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 01:50:31.784 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:52:32.152 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:55:31.823 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 01:57:32.191 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 02:00:31.862 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:02:32.230 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:05:31.901 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:07:32.270 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 02:10:31.940 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:12:32.309 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:15:31.979 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:17:32.348 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:20:32.018 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:22:32.387 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 02:25:32.057 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:27:32.427 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:30:32.096 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 02:32:32.465 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:35:32.135 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:37:32.505 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 02:40:32.174 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:42:32.544 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:45:32.213 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:47:32.583 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:50:32.252 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:52:32.621 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:55:32.291 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 02:57:32.661 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:00:32.330 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 03:02:32.699 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:05:32.369 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:07:32.738 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 03:10:32.408 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:12:32.778 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:15:32.447 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:17:32.817 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 03:20:32.485 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:22:32.856 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:25:32.525 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:27:32.895 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:30:32.563 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 03:32:32.934 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:35:32.602 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 03:37:32.973 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:40:32.641 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:42:33.012 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:45:32.689 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:47:33.051 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:50:32.727 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 03:52:33.090 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:55:32.766 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 03:57:33.130 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 04:00:32.806 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 04:02:33.168 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:05:32.845 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:07:33.208 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 04:10:32.884 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:12:33.247 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:15:32.923 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:17:33.285 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:20:32.961 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:22:33.324 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:25:33.000 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:27:33.363 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:30:33.039 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:32:33.402 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:35:33.078 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:37:33.442 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 04:40:33.117 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:42:33.481 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:45:33.156 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:47:33.519 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:50:33.194 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 04:52:33.558 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:55:33.233 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 04:57:33.597 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:00:33.272 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:02:33.636 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:05:33.311 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:07:33.675 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:10:33.350 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:12:33.714 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 05:15:33.388 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:17:33.753 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:20:33.427 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:22:33.791 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:25:33.466 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:27:33.830 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:30:33.505 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:32:33.869 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:35:33.544 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:37:33.909 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 05:40:33.582 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 05:42:33.948 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:45:33.621 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:47:33.986 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 05:50:33.659 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 05:52:34.025 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:55:33.698 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 05:57:34.064 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:00:33.737 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:02:34.103 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:05:33.775 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 06:07:34.142 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:10:33.814 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:12:34.181 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:15:33.853 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:17:34.219 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:20:33.891 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:22:34.258 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:25:33.931 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 06:27:34.297 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:30:33.970 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:32:34.336 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:35:34.008 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:37:34.375 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:40:34.047 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 06:42:34.413 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:45:34.085 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:47:34.452 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:50:34.124 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:52:34.491 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:55:34.163 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 06:57:34.531 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:00:34.202 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:02:34.570 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:05:34.241 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:07:34.609 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 07:10:34.279 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 07:12:34.648 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:15:34.318 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:17:34.686 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 07:20:34.356 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 07:22:34.725 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:25:34.395 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:27:34.765 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 07:30:34.433 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:32:34.804 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:35:34.472 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:37:34.843 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:40:34.511 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:42:34.883 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 07:45:34.550 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:47:34.921 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:50:34.590 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:52:34.961 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 07:55:34.629 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 07:57:35.000 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:00:34.668 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:02:35.040 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 08:05:34.707 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:07:35.079 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:10:34.746 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:12:35.118 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:15:34.785 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:17:35.156 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:20:34.823 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 08:22:35.195 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:25:34.862 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:27:35.234 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:30:34.900 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:32:35.273 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:35:34.939 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:37:35.311 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:40:34.977 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:42:35.350 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:45:35.016 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:47:35.389 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:50:35.055 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:52:35.428 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:55:35.094 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 08:57:35.468 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:00:35.133 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:02:35.507 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:05:35.172 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:07:35.545 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 09:10:35.211 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:12:35.584 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:15:35.250 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:17:35.623 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:20:35.289 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:22:35.662 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:25:35.328 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:27:35.701 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:30:35.366 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:32:35.740 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:35:35.405 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:37:35.779 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 09:40:35.443 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 09:42:35.818 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:45:35.482 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:47:35.857 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:50:35.520 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:52:35.896 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:55:35.559 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 09:57:35.935 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:00:35.598 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:02:35.974 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 10:05:35.637 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:07:36.013 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:10:35.676 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:12:36.053 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:15:35.716 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 10:17:36.091 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 10:20:35.755 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:22:36.130 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:25:35.794 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:27:36.169 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:30:35.833 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:32:36.208 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:35:35.872 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:37:36.247 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:40:35.910 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 10:42:36.286 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:45:35.949 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:47:36.325 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:50:35.988 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 10:52:36.364 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:55:36.027 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 10:57:36.404 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:00:36.066 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:02:36.442 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:05:36.104 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:07:36.481 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:10:36.143 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:12:36.519 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 11:15:36.181 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 11:17:36.559 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:20:36.221 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 11:22:36.598 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:25:36.260 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:27:36.637 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 11:30:36.298 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:32:36.676 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:35:36.337 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:37:36.715 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 11:40:36.376 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:42:36.754 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:45:36.415 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:47:36.793 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:50:36.454 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:52:36.832 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:55:36.492 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 11:57:36.871 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:00:36.531 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:02:36.912 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 12:05:36.570 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 12:07:36.950 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:10:36.609 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:12:36.989 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 12:15:36.648 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:17:37.028 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:20:36.687 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 12:22:37.067 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:25:36.726 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:27:37.106 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:30:36.765 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:32:37.144 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:35:36.803 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:37:37.183 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:40:36.843 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:42:37.222 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 12:45:36.882 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:47:37.261 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:50:36.921 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:52:37.300 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 12:55:36.959 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 12:57:37.339 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:00:36.998 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:02:37.378 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 13:05:37.037 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:07:37.417 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:10:37.076 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:12:37.456 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:15:37.116 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:17:37.495 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:20:37.154 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:22:37.534 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:25:37.193 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:27:37.573 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:30:37.232 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:32:37.612 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 13:35:37.270 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:37:37.652 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:40:37.309 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:42:37.691 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 13:45:37.348 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:47:37.731 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 13:50:37.387 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 13:52:37.771 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 13:55:37.426 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 13:57:37.809 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:00:37.466 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 14:02:37.848 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:05:37.505 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:07:37.887 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 14:10:37.544 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:12:37.927 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:15:37.582 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:17:37.966 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:20:37.630 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 14:22:38.004 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 14:25:37.670 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:27:38.043 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 14:29:08.565 [conn3625] build index dicomdb.fs.files { filename: 1, uploadDate: 1 } Sun Nov 24 14:29:08.580 [conn3625] build index done. scanned 0 total records. 0.014 secs Sun Nov 24 14:29:08.610 [conn3625] build index dicomdb.fs.chunks { files_id: 1, n: 1 } Sun Nov 24 14:29:08.613 [conn3625] build index done. scanned 0 total records. 0.002 secs Sun Nov 24 14:30:37.709 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:32:38.081 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:34:45.109 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:45.110 [conn10] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:34:46.298 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:2229737 1115ms Sun Nov 24 14:34:46.305 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:46.306 [conn10] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:34:47.233 [conn3626] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:359 731ms Sun Nov 24 14:34:47.233 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:233 726ms Sun Nov 24 14:34:47.256 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:47.257 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:47.257 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:34:47.259 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:47.259 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:47.259 [conn10] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:34:47.286 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:34:47.286 [conn10] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:34:47.287 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:34:47.287 [conn7] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:34:47.287 [conn10] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:34:47.288 [conn10] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:34:47.309 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:47.310 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:47.335 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:34:47.335 [conn10] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:34:47.335 [LockPinger] creating distributed lock ping thread for node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020 and process node40.clus.cci.emory.edu:27001:1385321687:2071045908 (sleeping for 30000ms) Sun Nov 24 14:34:47.335 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:34:47.335 [conn10] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:34:47.336 [conn7] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:34:47.336 [conn10] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:34:51.100 [conn10] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0 } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn10:1345868337", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385321690564), why: "split-{ files_id: MinKey }", ts: ObjectId('529254da18c2315fef764a86') } } gle1: { updatedExisting: false, n: 0, connectionId: 29, waited: 19, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385321690000|5, connectionId: 62, waited: 42, err: null, ok: 1.0 } Sun Nov 24 14:34:51.100 [conn7] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0 } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385321690529), why: "split-{ files_id: MinKey }", ts: ObjectId('529254da18c2315fef764a85') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385321690000|6, connectionId: 28, waited: 19, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, connectionId: 61, waited: 42, err: null, ok: 1.0 } Sun Nov 24 14:34:51.102 [conn7] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:34:51.102 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:34:51.102 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:351 3843ms Sun Nov 24 14:34:51.104 [conn10] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:34:51.141 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.142 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.142 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:34:51.144 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:51.159 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.159 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.159 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:34:51.161 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:51.171 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.171 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.171 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:34:51.173 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:51.183 [conn7] request split points lookup for chunk dicomdb.fs.files { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.183 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.files { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.183 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:51.397 [LockPinger] trying to delete 1 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:34:51.398 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529254da18c2315fef764a86 Sun Nov 24 14:34:51.400 [conn10] splitChunk accepted at version 1|0||528f906bc868398bb45de3b3 Sun Nov 24 14:34:51.503 [LockPinger] handled late remove of old distributed lock with ts 529254da18c2315fef764a85 Sun Nov 24 14:34:51.512 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:34:51-529254db18c2315fef764a88", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321691512), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: MinKey }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('528f906bc868398bb45de3b3') }, right: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('528f906bc868398bb45de3b3') } } } Sun Nov 24 14:34:51.679 [conn7] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529254db18c2315fef764a87 Sun Nov 24 14:34:51.681 [conn7] splitChunk accepted at version 1|0||528f906ec868398bb45de3b5 Sun Nov 24 14:34:51.785 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:34:51-529254db18c2315fef764a89", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385321691785), what: "split", ns: "dicomdb.fs.files", details: { before: { min: { _id: MinKey }, max: { _id: MaxKey }, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('528f906ec868398bb45de3b5') }, right: { min: { _id: ObjectId('529254d50cf2e381b51da43d') }, max: { _id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('528f906ec868398bb45de3b5') } } } Sun Nov 24 14:34:51.961 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:34:51.961 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:103 reslen:37 4699ms Sun Nov 24 14:34:51.973 [conn10] request split points lookup for chunk dicomdb.fs.files { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.973 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.files { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:51.974 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:52.003 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:34:52.003 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:34:52.005 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254db0cf2e381b51da44e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:52.032 [conn7] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:34:52.032 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:86 reslen:37 848ms Sun Nov 24 14:34:52.078 [conn7] request split points lookup for chunk dicomdb.dicom { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:52.095 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:34:52.095 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:34:52.097 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254dc0cf2e381b51da453') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:52.208 [conn10] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529254da18c2315fef764a86') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn10:1345868337", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385321692006), why: "split-{ files_id: ObjectId('529254d50cf2e381b51da43d') }", ts: ObjectId('529254dc18c2315fef764a8a') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385321691000|13, connectionId: 28, waited: 33, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385321692000|1, connectionId: 61, waited: 31, err: null, ok: 1.0 } Sun Nov 24 14:34:52.208 [conn7] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529254da18c2315fef764a86') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385321692098), why: "split-{ files_id: ObjectId('529254d50cf2e381b51da43d') }", ts: ObjectId('529254dc18c2315fef764a8b') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385321692000|1, connectionId: 29, waited: 34, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385321691000|14, connectionId: 62, waited: 31, err: null, ok: 1.0 } Sun Nov 24 14:34:52.209 [conn10] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:34:52.209 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:34:52.210 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254db0cf2e381b51da44e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:380 204ms Sun Nov 24 14:34:52.212 [conn7] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:34:52.223 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:34:52.223 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:34:52.226 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254dc0cf2e381b51da453') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:52.245 [conn10] request split points lookup for chunk dicomdb.dicom { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:52.245 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : MinKey } -->> { : MaxKey } Sun Nov 24 14:34:52.246 [conn10] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:52.490 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529254dc18c2315fef764a8b Sun Nov 24 14:34:52.492 [conn7] splitChunk accepted at version 1|2||528f906bc868398bb45de3b3 Sun Nov 24 14:34:52.596 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:34:52-529254dc18c2315fef764a8d", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385321692596), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529254dc0cf2e381b51da453') }, lastmod: Timestamp 1000|3, lastmodEpoch: ObjectId('528f906bc868398bb45de3b3') }, right: { min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|4, lastmodEpoch: ObjectId('528f906bc868398bb45de3b3') } } } Sun Nov 24 14:34:52.702 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529254dc18c2315fef764a8c Sun Nov 24 14:34:52.704 [conn10] splitChunk accepted at version 1|0||528f906dc868398bb45de3b4 Sun Nov 24 14:34:52.843 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:34:52.843 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254dc0cf2e381b51da453') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:74 reslen:109 745ms Sun Nov 24 14:34:52.843 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:34:52-529254dc18c2315fef764a8e", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321692843), what: "split", ns: "dicomdb.dicom", details: { before: { min: { files_id: MinKey }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('528f906dc868398bb45de3b4') }, right: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('528f906dc868398bb45de3b4') } } } Sun Nov 24 14:34:52.899 [conn7] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: false, waitForDelete: false } Sun Nov 24 14:34:52.984 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:34:52.984 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254d50cf2e381b51da43d') } ], shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:94 reslen:37 737ms Sun Nov 24 14:34:53.090 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529254dc18c2315fef764a8f Sun Nov 24 14:34:53.090 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:34:53-529254dd18c2315fef764a90", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385321693090), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:34:53.286 [conn3626] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:162 183ms Sun Nov 24 14:34:53.481 [conn7] moveChunk request accepted at version 1|4||528f906bc868398bb45de3b3 Sun Nov 24 14:34:53.652 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:448933 225ms Sun Nov 24 14:34:53.652 [conn7] moveChunk number of documents: 45 Sun Nov 24 14:34:53.653 [conn7] starting new replica set monitor for replica set dicom10 with seed of node40.clus.cci.emory.edu:27010 Sun Nov 24 14:34:53.653 [conn7] successfully connected to seed node40.clus.cci.emory.edu:27010 for replica set dicom10 Sun Nov 24 14:34:53.654 [conn7] changing hosts to { 0: "node40.clus.cci.emory.edu:27010" } from dicom10/ Sun Nov 24 14:34:53.654 [conn7] trying to add new host node40.clus.cci.emory.edu:27010 to replica set dicom10 Sun Nov 24 14:34:53.654 [conn7] successfully connected to new host node40.clus.cci.emory.edu:27010 in replica set dicom10 Sun Nov 24 14:34:53.657 [conn7] Primary for replica set dicom10 changed to node40.clus.cci.emory.edu:27010 Sun Nov 24 14:34:53.657 [conn7] replica set monitor for replica set dicom10 started, address is dicom10/node40.clus.cci.emory.edu:27010 Sun Nov 24 14:34:53.657 [ReplicaSetMonitorWatcher] starting Sun Nov 24 14:34:53.661 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:53.664 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:53.669 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:53.678 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:53.695 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:53.728 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:53.793 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:53.922 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:54.179 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:54.692 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:55.038 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1427954 714ms Sun Nov 24 14:34:55.669 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1035632 518ms Sun Nov 24 14:34:55.717 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:56.100 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:635520 318ms Sun Nov 24 14:34:56.599 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:776937 389ms Sun Nov 24 14:34:56.599 [conn3630] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:59 reslen:51 385ms Sun Nov 24 14:34:56.744 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:56.867 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:322524 161ms Sun Nov 24 14:34:56.872 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3661 reslen:1052358 120ms Sun Nov 24 14:34:57.262 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:571699 286ms Sun Nov 24 14:34:57.266 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3311 reslen:1052358 202ms Sun Nov 24 14:34:57.744 [conn3626] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:176 371ms Sun Nov 24 14:34:57.769 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:58.113 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:502441 251ms Sun Nov 24 14:34:58.342 [conn3626] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:216 124ms Sun Nov 24 14:34:58.395 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.2, filling with zeroes... Sun Nov 24 14:34:58.794 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:34:59.272 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:802273 401ms Sun Nov 24 14:34:59.277 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:4908 reslen:1155767 405ms Sun Nov 24 14:34:59.379 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:34:59.379 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:34:59.382 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254e30cf2e381b51da68d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:34:59.715 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:659414 329ms Sun Nov 24 14:34:59.720 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3695 reslen:1207553 308ms Sun Nov 24 14:34:59.820 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:00.163 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:670109 335ms Sun Nov 24 14:35:00.583 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:616142 308ms Sun Nov 24 14:35:00.847 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:00.896 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:00.897 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:00.899 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254e40cf2e381b51da700') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:01.188 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:389150 194ms Sun Nov 24 14:35:01.821 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1032530 516ms Sun Nov 24 14:35:01.825 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:2580 reslen:1052522 289ms Sun Nov 24 14:35:01.872 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:02.320 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:768765 384ms Sun Nov 24 14:35:02.371 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:02.372 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:02.374 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254e60cf2e381b51da773') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:02.713 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.2, size: 256MB, took 4.318 secs Sun Nov 24 14:35:02.717 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:566537 283ms Sun Nov 24 14:35:02.720 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:2121 reslen:1052522 282ms Sun Nov 24 14:35:02.897 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:03.548 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:637111 318ms Sun Nov 24 14:35:03.552 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3440 reslen:1052374 263ms Sun Nov 24 14:35:03.592 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:03.592 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:03.594 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254e70cf2e381b51da7e6') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:03.895 [conn3626] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:470413 235ms Sun Nov 24 14:35:03.922 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:04.358 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:691975 346ms Sun Nov 24 14:35:04.362 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3385 reslen:1052218 346ms Sun Nov 24 14:35:04.447 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:04.448 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:04.450 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254e80cf2e381b51da859') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:04.464 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:04.465 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:04.467 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254e80cf2e381b51da859') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:04.770 [conn3626] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:195 300ms Sun Nov 24 14:35:04.947 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:05.199 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:632297 316ms Sun Nov 24 14:35:05.610 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:593175 297ms Sun Nov 24 14:35:05.717 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:05.718 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:05.720 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254e90cf2e381b51da8cb') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:05.972 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:06.019 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:589674 295ms Sun Nov 24 14:35:06.022 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:2378 reslen:1052222 293ms Sun Nov 24 14:35:06.354 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:445739 223ms Sun Nov 24 14:35:06.932 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:919016 460ms Sun Nov 24 14:35:06.935 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:2105 reslen:1052226 405ms Sun Nov 24 14:35:06.967 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:06.968 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:06.970 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254ea0cf2e381b51da93e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:06.998 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:07.736 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:500819 250ms Sun Nov 24 14:35:07.816 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:07.816 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:07.817 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254eb0cf2e381b51da9b1') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:08.024 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:08.059 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:414962 208ms Sun Nov 24 14:35:08.064 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:4043 reslen:1261230 148ms Sun Nov 24 14:35:08.422 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:493873 247ms Sun Nov 24 14:35:08.422 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:91 reslen:3137 255ms Sun Nov 24 14:35:08.427 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3893 reslen:1209502 252ms Sun Nov 24 14:35:08.472 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:35:08.472 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:35:08.473 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:08-529254ec18c2315fef764a91", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385321708473), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, step1 of 6: 0, note: "aborted" } } Sun Nov 24 14:35:08.533 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:08.533 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:08.536 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254ec0cf2e381b51daa1c') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:08.862 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:644450 322ms Sun Nov 24 14:35:08.866 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3388 reslen:1052586 318ms Sun Nov 24 14:35:08.871 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:08.872 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:08.874 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254ec0cf2e381b51daa1c') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:09.050 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:09.111 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:79 638ms Sun Nov 24 14:35:09.114 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:35:09.115 [conn15] received moveChunk request: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:35:09.115 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:09-529254ed18c2315fef764a92", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385321709115), what: "moveChunk.from", ns: "dicomdb.dicom", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, step1 of 6: 1, note: "aborted" } } Sun Nov 24 14:35:09.144 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.3, filling with zeroes... Sun Nov 24 14:35:09.446 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:531138 266ms Sun Nov 24 14:35:09.451 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3532 reslen:1052586 202ms Sun Nov 24 14:35:09.533 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:09.534 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:09.536 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254ed0cf2e381b51daa8f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:09.919 [conn3626] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:720352 360ms Sun Nov 24 14:35:09.923 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3397 reslen:1052582 255ms Sun Nov 24 14:35:10.076 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:10.318 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:569747 284ms Sun Nov 24 14:35:10.322 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3340 reslen:1052594 284ms Sun Nov 24 14:35:10.723 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:578401 289ms Sun Nov 24 14:35:10.730 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3542 reslen:1052594 298ms Sun Nov 24 14:35:10.751 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:10.752 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:10.753 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254ee0cf2e381b51dab02') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:11.102 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:11.247 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:79 2132ms Sun Nov 24 14:35:11.250 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:35:11.251 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:35:11.251 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:11-529254ef18c2315fef764a93", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385321711251), what: "moveChunk.from", ns: "dicomdb.fs.files", details: { min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, step1 of 6: 1, note: "aborted" } } Sun Nov 24 14:35:11.858 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3358 reslen:1052594 407ms Sun Nov 24 14:35:12.217 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:12.303 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:663776 332ms Sun Nov 24 14:35:12.307 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3383 reslen:1052594 171ms Sun Nov 24 14:35:12.313 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:12.314 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:12.316 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254ef0cf2e381b51dab76') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:12.848 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:860986 431ms Sun Nov 24 14:35:12.852 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3422 reslen:1052590 328ms Sun Nov 24 14:35:13.203 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:482696 241ms Sun Nov 24 14:35:13.207 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3525 reslen:1052590 251ms Sun Nov 24 14:35:13.244 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:13.269 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:13.270 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:13.273 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254f10cf2e381b51dabe9') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:13.288 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:13.289 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:13.291 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254f10cf2e381b51dabe9') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:13.670 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:713518 357ms Sun Nov 24 14:35:13.674 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3385 reslen:1052590 148ms Sun Nov 24 14:35:14.130 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:691062 345ms Sun Nov 24 14:35:14.135 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3371 reslen:1052594 204ms Sun Nov 24 14:35:14.238 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:14.238 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:14.240 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254f20cf2e381b51dac5c') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:14.270 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:14.686 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:874677 437ms Sun Nov 24 14:35:14.691 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3527 reslen:1052594 436ms Sun Nov 24 14:35:15.104 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:609820 305ms Sun Nov 24 14:35:15.110 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:4158 reslen:1305766 140ms Sun Nov 24 14:35:15.296 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:15.493 [conn3626] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:542978 272ms Sun Nov 24 14:35:15.497 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3343 reslen:1052554 222ms Sun Nov 24 14:35:15.521 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:15.521 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529254dc0cf2e381b51da453') } -->> { : MaxKey } Sun Nov 24 14:35:15.523 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529254f30cf2e381b51daccf') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:35:15.893 [conn3626] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:230 287ms Sun Nov 24 14:35:16.322 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:16.418 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:3457 reslen:1052574 185ms Sun Nov 24 14:35:16.572 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.3, size: 512MB, took 7.427 secs Sun Nov 24 14:35:17.348 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:18.374 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:19.400 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:20.426 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:21.452 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:22.478 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:22.478 [conn7] moveChunk setting version to: 2|0||528f906bc868398bb45de3b3 Sun Nov 24 14:35:22.486 [conn7] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 45, clonedBytes: 8152550, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:35:22.486 [conn7] moveChunk updating self version to: 2|1||528f906bc868398bb45de3b3 through { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:35:22.486 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:35:22.487 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:35:22.487 [conn7] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:35:26.526 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:79 15276ms Sun Nov 24 14:35:26.526 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:26-529254fe18c2315fef764a94", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385321726526), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:35:26.992 [conn7] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:26.992 [LockPinger] trying to delete 1 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:35:26.992 [conn7] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:26.992 [conn7] forking for cleanup of chunk data Sun Nov 24 14:35:26.992 [conn7] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:26.992 [conn7] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:26.992 [cleanupOldData-529254fe18c2315fef764a95] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529254dc0cf2e381b51da453') } -> { files_id: MaxKey }, # cursors remaining: 0 Sun Nov 24 14:35:27.013 [cleanupOldData-529254fe18c2315fef764a95] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529254dc0cf2e381b51da453') } -> { files_id: MaxKey } Sun Nov 24 14:35:27.014 [cleanupOldData-529254fe18c2315fef764a95] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529254dc0cf2e381b51da453') } -> { files_id: MaxKey } Sun Nov 24 14:35:27.168 [LockPinger] handled late remove of old distributed lock with ts 529254dc18c2315fef764a8a Sun Nov 24 14:35:27.168 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:35:27.168 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:27-529254ff18c2315fef764a96", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385321727168), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, step1 of 6: 1, step2 of 6: 581, step3 of 6: 177, step4 of 6: 28819, step5 of 6: 4514, step6 of 6: 0 } } Sun Nov 24 14:35:27.179 [cleanupOldData-529254fe18c2315fef764a95] moveChunk deleted 1346 documents for dicomdb.fs.chunks from { files_id: ObjectId('529254dc0cf2e381b51da453') } -> { files_id: MaxKey } Sun Nov 24 14:35:27.449 [conn7] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: ObjectId('529254dc0cf2e381b51da453') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529254dc0cf2e381b51da453')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: false, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:109 r:314 w:33 reslen:37 34558ms Sun Nov 24 14:35:33.569 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:35:33.571 [conn10] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:35:33.572 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:33-5292550518c2315fef764a97", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321733572), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, step1 of 6: 1, note: "aborted" } } Sun Nov 24 14:35:33.969 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:423 399ms Sun Nov 24 14:35:33.972 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:35:33.973 [conn10] received moveChunk request: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:35:34.182 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292550518c2315fef764a98 Sun Nov 24 14:35:34.182 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:34-5292550618c2315fef764a99", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321734182), what: "moveChunk.start", ns: "dicomdb.dicom", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:35:34.254 [conn10] moveChunk request accepted at version 1|2||528f906dc868398bb45de3b4 Sun Nov 24 14:35:34.255 [conn10] moveChunk number of documents: 0 Sun Nov 24 14:35:34.257 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.260 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.265 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.274 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.274 [conn10] moveChunk setting version to: 2|0||528f906dc868398bb45de3b4 Sun Nov 24 14:35:34.282 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:35:34.282 [conn10] moveChunk updating self version to: 2|1||528f906dc868398bb45de3b4 through { files_id: ObjectId('529254d50cf2e381b51da43d') } -> { files_id: MaxKey } for collection 'dicomdb.dicom' Sun Nov 24 14:35:34.431 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:34-5292550618c2315fef764a9a", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321734431), what: "moveChunk.commit", ns: "dicomdb.dicom", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:35:34.503 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:34.503 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:34.503 [conn10] forking for cleanup of chunk data Sun Nov 24 14:35:34.503 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:34.503 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:34.503 [cleanupOldData-5292550618c2315fef764a9b] (start) waiting to cleanup dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') }, # cursors remaining: 0 Sun Nov 24 14:35:34.523 [cleanupOldData-5292550618c2315fef764a9b] waiting to remove documents for dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:34.523 [cleanupOldData-5292550618c2315fef764a9b] moveChunk starting delete for: dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:34.524 [cleanupOldData-5292550618c2315fef764a9b] moveChunk deleted 0 documents for dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:34.574 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:35:34.574 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:34-5292550618c2315fef764a9c", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321734574), what: "moveChunk.from", ns: "dicomdb.dicom", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, step1 of 6: 1, step2 of 6: 281, step3 of 6: 0, step4 of 6: 18, step5 of 6: 229, step6 of 6: 0 } } Sun Nov 24 14:35:34.645 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:39 r:157 w:25 reslen:37 672ms Sun Nov 24 14:35:34.646 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:35:34.647 [conn10] received moveChunk request: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:35:34.822 [conn10] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292550618c2315fef764a9d Sun Nov 24 14:35:34.822 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:34-5292550618c2315fef764a9e", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321734822), what: "moveChunk.start", ns: "dicomdb.fs.files", details: { min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:35:34.928 [conn10] moveChunk request accepted at version 1|2||528f906ec868398bb45de3b5 Sun Nov 24 14:35:34.928 [conn10] moveChunk number of documents: 0 Sun Nov 24 14:35:34.931 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.934 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.939 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { _id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.948 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { _id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.965 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { _id: 1.0 }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:34.965 [conn10] moveChunk setting version to: 2|0||528f906ec868398bb45de3b5 Sun Nov 24 14:35:34.973 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { _id: 1.0 }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:35:34.973 [conn10] moveChunk updating self version to: 2|1||528f906ec868398bb45de3b5 through { _id: ObjectId('529254d50cf2e381b51da43d') } -> { _id: MaxKey } for collection 'dicomdb.fs.files' Sun Nov 24 14:35:35.071 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:35-5292550718c2315fef764a9f", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321735071), what: "moveChunk.commit", ns: "dicomdb.fs.files", details: { min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:35:35.318 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:35.318 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:35.318 [conn10] forking for cleanup of chunk data Sun Nov 24 14:35:35.318 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:35.318 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:35.318 [cleanupOldData-5292550718c2315fef764aa0] (start) waiting to cleanup dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529254d50cf2e381b51da43d') }, # cursors remaining: 0 Sun Nov 24 14:35:35.339 [cleanupOldData-5292550718c2315fef764aa0] waiting to remove documents for dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:35.340 [cleanupOldData-5292550718c2315fef764aa0] moveChunk starting delete for: dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:35.340 [cleanupOldData-5292550718c2315fef764aa0] moveChunk deleted 0 documents for dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:35.424 [conn10] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:35:35.424 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:35-5292550718c2315fef764aa1", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321735424), what: "moveChunk.from", ns: "dicomdb.fs.files", details: { min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, step1 of 6: 1, step2 of 6: 281, step3 of 6: 0, step4 of 6: 36, step5 of 6: 353, step6 of 6: 0 } } Sun Nov 24 14:35:35.495 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { _id: MinKey }, max: { _id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:39 r:98 w:23 reslen:37 848ms Sun Nov 24 14:35:36.921 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:35:36.922 [conn10] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:35:37.065 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292550818c2315fef764aa2 Sun Nov 24 14:35:37.065 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:37-5292550918c2315fef764aa3", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321737065), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, from: "dicom1", to: "dicom12" } } Sun Nov 24 14:35:37.245 [conn10] moveChunk request accepted at version 2|1||528f906bc868398bb45de3b3 Sun Nov 24 14:35:37.245 [conn10] moveChunk number of documents: 0 Sun Nov 24 14:35:37.245 [conn10] starting new replica set monitor for replica set dicom12 with seed of node40.clus.cci.emory.edu:27012 Sun Nov 24 14:35:37.246 [conn10] successfully connected to seed node40.clus.cci.emory.edu:27012 for replica set dicom12 Sun Nov 24 14:35:37.246 [conn10] changing hosts to { 0: "node40.clus.cci.emory.edu:27012" } from dicom12/ Sun Nov 24 14:35:37.247 [conn10] trying to add new host node40.clus.cci.emory.edu:27012 to replica set dicom12 Sun Nov 24 14:35:37.247 [conn10] successfully connected to new host node40.clus.cci.emory.edu:27012 in replica set dicom12 Sun Nov 24 14:35:37.249 [conn10] Primary for replica set dicom12 changed to node40.clus.cci.emory.edu:27012 Sun Nov 24 14:35:37.250 [conn10] replica set monitor for replica set dicom12 started, address is dicom12/node40.clus.cci.emory.edu:27012 Sun Nov 24 14:35:37.253 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.256 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.261 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.270 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.287 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.320 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.385 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.514 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:37.747 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 14:35:37.771 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:38.284 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:39.309 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:35:39.309 [conn10] moveChunk setting version to: 4|0||528f906bc868398bb45de3b3 Sun Nov 24 14:35:39.320 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:35:39.320 [conn10] moveChunk updating self version to: 4|1||528f906bc868398bb45de3b3 through { files_id: ObjectId('529254d50cf2e381b51da43d') } -> { files_id: ObjectId('529254dc0cf2e381b51da453') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:35:40.655 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:40-5292550c18c2315fef764aa4", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321740655), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, from: "dicom1", to: "dicom12" } } Sun Nov 24 14:35:40.773 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:40.773 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:40.773 [conn10] forking for cleanup of chunk data Sun Nov 24 14:35:40.773 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:35:40.773 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:35:40.773 [cleanupOldData-5292550c18c2315fef764aa5] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') }, # cursors remaining: 0 Sun Nov 24 14:35:40.794 [cleanupOldData-5292550c18c2315fef764aa5] waiting to remove documents for dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:40.794 [cleanupOldData-5292550c18c2315fef764aa5] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:40.795 [cleanupOldData-5292550c18c2315fef764aa5] moveChunk deleted 0 documents for dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529254d50cf2e381b51da43d') } Sun Nov 24 14:35:40.917 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:35:40.917 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:35:40-5292550c18c2315fef764aa6", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385321740917), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, step1 of 6: 1, step2 of 6: 322, step3 of 6: 6, step4 of 6: 2057, step5 of 6: 1464, step6 of 6: 0 } } Sun Nov 24 14:35:41.026 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: MinKey }, max: { files_id: ObjectId('529254d50cf2e381b51da43d') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:38 r:139 w:28 reslen:37 4105ms Sun Nov 24 14:37:38.121 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:39:36.213 [LockPinger] cluster node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020 pinged successfully at Sun Nov 24 14:39:35 2013 by distributed lock pinger 'node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020/node40.clus.cci.emory.edu:27001:1385321687:2071045908', sleeping for 30000ms Sun Nov 24 14:40:37.786 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:42:38.160 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300039ms Sun Nov 24 14:43:45.211 [conn7] request split points lookup for chunk dicomdb.dicom { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:43:45.212 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:43:45.214 [conn7] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529256f10cf278517e778823') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:43:45.218 [conn10] request split points lookup for chunk dicomdb.dicom { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:43:45.272 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:43:45.273 [conn10] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529256f10cf278517e778823') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:43:45.634 [conn10] could not acquire lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:43:45.634 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:43:45.635 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529256f10cf278517e778823') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:376 361ms Sun Nov 24 14:43:46.094 [conn7] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529256f118c2315fef764aa7 Sun Nov 24 14:43:46.096 [conn7] splitChunk accepted at version 2|1||528f906dc868398bb45de3b4 Sun Nov 24 14:43:46.358 [conn10] request split points lookup for chunk dicomdb.dicom { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:43:46.359 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : ObjectId('529254d50cf2e381b51da43d') } -->> { : MaxKey } Sun Nov 24 14:43:46.360 [conn10] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529256f20cf278517e778886') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:43:46.777 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:43:46-529256f218c2315fef764aa9", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322226777), what: "split", ns: "dicomdb.dicom", details: { before: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, lastmod: Timestamp 2000|1, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('528f906dc868398bb45de3b4') }, right: { min: { files_id: ObjectId('529256f10cf278517e778823') }, max: { files_id: MaxKey }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('528f906dc868398bb45de3b4') } } } Sun Nov 24 14:43:47.496 [conn7] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:43:47.496 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529256f10cf278517e778823') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:111 reslen:37 2282ms Sun Nov 24 14:43:51.035 [conn7] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:43:51.036 [conn7] received moveChunk request: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:43:51.253 [conn7] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529256f718c2315fef764aaa Sun Nov 24 14:43:51.253 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:43:51-529256f718c2315fef764aab", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322231253), what: "moveChunk.start", ns: "dicomdb.dicom", details: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:43:51.290 [conn7] moveChunk request accepted at version 2|3||528f906dc868398bb45de3b4 Sun Nov 24 14:43:51.292 [conn7] moveChunk number of documents: 627 Sun Nov 24 14:43:51.292 [conn7] starting new replica set monitor for replica set dicom11 with seed of node40.clus.cci.emory.edu:27011 Sun Nov 24 14:43:51.293 [conn7] successfully connected to seed node40.clus.cci.emory.edu:27011 for replica set dicom11 Sun Nov 24 14:43:51.293 [conn7] changing hosts to { 0: "node40.clus.cci.emory.edu:27011" } from dicom11/ Sun Nov 24 14:43:51.293 [conn7] trying to add new host node40.clus.cci.emory.edu:27011 to replica set dicom11 Sun Nov 24 14:43:51.294 [conn7] successfully connected to new host node40.clus.cci.emory.edu:27011 in replica set dicom11 Sun Nov 24 14:43:51.296 [conn7] Primary for replica set dicom11 changed to node40.clus.cci.emory.edu:27011 Sun Nov 24 14:43:51.296 [conn7] replica set monitor for replica set dicom11 started, address is dicom11/node40.clus.cci.emory.edu:27011 Sun Nov 24 14:43:51.299 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:43:51.303 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:43:51.308 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:43:51.317 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 15, clonedBytes: 2908, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:43:51.334 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 197, clonedBytes: 38267, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:43:51.367 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 325, clonedBytes: 63205, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:43:51.433 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 627, clonedBytes: 580089, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:43:51.433 [conn7] moveChunk setting version to: 3|0||528f906dc868398bb45de3b4 Sun Nov 24 14:43:51.447 [conn7] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 627, clonedBytes: 580089, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:43:51.447 [conn7] moveChunk updating self version to: 3|1||528f906dc868398bb45de3b4 through { files_id: ObjectId('529256f10cf278517e778823') } -> { files_id: MaxKey } for collection 'dicomdb.dicom' Sun Nov 24 14:43:51.648 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:43:51-529256f718c2315fef764aac", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322231648), what: "moveChunk.commit", ns: "dicomdb.dicom", details: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:43:51.755 [conn7] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:43:51.755 [conn7] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:43:51.755 [conn7] forking for cleanup of chunk data Sun Nov 24 14:43:51.755 [conn7] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:43:51.755 [conn7] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:43:51.755 [cleanupOldData-529256f718c2315fef764aad] (start) waiting to cleanup dicomdb.dicom from { files_id: ObjectId('529254d50cf2e381b51da43d') } -> { files_id: ObjectId('529256f10cf278517e778823') }, # cursors remaining: 0 Sun Nov 24 14:43:51.777 [cleanupOldData-529256f718c2315fef764aad] waiting to remove documents for dicomdb.dicom from { files_id: ObjectId('529254d50cf2e381b51da43d') } -> { files_id: ObjectId('529256f10cf278517e778823') } Sun Nov 24 14:43:51.777 [cleanupOldData-529256f718c2315fef764aad] moveChunk starting delete for: dicomdb.dicom from { files_id: ObjectId('529254d50cf2e381b51da43d') } -> { files_id: ObjectId('529256f10cf278517e778823') } Sun Nov 24 14:43:51.826 [conn7] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:43:51.826 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:43:51-529256f718c2315fef764aae", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322231826), what: "moveChunk.from", ns: "dicomdb.dicom", details: { min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, step1 of 6: 1, step2 of 6: 254, step3 of 6: 6, step4 of 6: 136, step5 of 6: 321, step6 of 6: 0 } } Sun Nov 24 14:43:51.852 [cleanupOldData-529256f718c2315fef764aad] moveChunk deleted 627 documents for dicomdb.dicom from { files_id: ObjectId('529254d50cf2e381b51da43d') } -> { files_id: ObjectId('529256f10cf278517e778823') } Sun Nov 24 14:43:51.897 [conn7] command admin.$cmd command: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { files_id: ObjectId('529254d50cf2e381b51da43d') }, max: { files_id: ObjectId('529256f10cf278517e778823') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_ObjectId('529254d50cf2e381b51da43d')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:45 r:1682 w:26 reslen:37 862ms Sun Nov 24 14:44:40.660 [LockPinger] cluster node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020 pinged successfully at Sun Nov 24 14:44:40 2013 by distributed lock pinger 'node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020/node40.clus.cci.emory.edu:27001:1385321687:2071045908', sleeping for 30000ms Sun Nov 24 14:45:37.825 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:45:49.609 [conn7] wiping data for: dicomdb.dicom Sun Nov 24 14:45:50.526 [conn7] wiping data for: dicomdb.fs.chunks Sun Nov 24 14:45:51.450 [conn7] wiping data for: dicomdb.fs.files Sun Nov 24 14:45:52.051 [conn3639] dropDatabase dicomdb starting Sun Nov 24 14:45:52.581 [conn3639] removeJournalFiles Sun Nov 24 14:45:53.036 [conn3639] dropDatabase dicomdb finished Sun Nov 24 14:45:53.037 [conn3639] command dicomdb.$cmd command: { dropDatabase: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:985268 reslen:58 985ms Sun Nov 24 14:45:53.094 [conn3639] dropDatabase dicomdb starting Sun Nov 24 14:45:53.129 [conn3639] removeJournalFiles Sun Nov 24 14:45:53.134 [conn3639] dropDatabase dicomdb finished Sun Nov 24 14:47:38.198 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300037ms Sun Nov 24 14:49:45.095 [LockPinger] cluster node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020 pinged successfully at Sun Nov 24 14:49:44 2013 by distributed lock pinger 'node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020/node40.clus.cci.emory.edu:27001:1385321687:2071045908', sleeping for 30000ms Sun Nov 24 14:50:37.865 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:51:15.492 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.ns, filling with zeroes... Sun Nov 24 14:51:15.492 [FileAllocator] creating directory /shared/mongodb/data/db/dicom/dicom1/dicomdb/_tmp Sun Nov 24 14:51:15.808 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.ns, size: 16MB, took 0.241 secs Sun Nov 24 14:51:15.808 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.0, filling with zeroes... Sun Nov 24 14:51:16.722 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.0, size: 64MB, took 0.912 secs Sun Nov 24 14:51:16.722 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.1, filling with zeroes... Sun Nov 24 14:51:16.726 [conn10] build index dicomdb.fs.chunks { _id: 1 } Sun Nov 24 14:51:16.728 [conn10] build index done. scanned 0 total records. 0.002 secs Sun Nov 24 14:51:16.728 [conn10] info: creating collection dicomdb.fs.chunks on add index Sun Nov 24 14:51:16.728 [conn10] build index dicomdb.fs.chunks { files_id: 1.0 } Sun Nov 24 14:51:16.777 [conn10] build index done. scanned 0 total records. 0.048 secs Sun Nov 24 14:51:16.777 [conn10] insert dicomdb.system.indexes ninserted:1 keyUpdates:0 locks(micros) w:1285863 1285ms Sun Nov 24 14:51:18.828 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.1, size: 128MB, took 2.104 secs Sun Nov 24 14:51:19.991 [conn3626] no current chunk manager found for this shard, will initialize Sun Nov 24 14:51:20.031 [conn10] build index dicomdb.dicom { _id: 1 } Sun Nov 24 14:51:20.033 [conn10] build index done. scanned 0 total records. 0.002 secs Sun Nov 24 14:51:20.033 [conn10] info: creating collection dicomdb.dicom on add index Sun Nov 24 14:51:20.033 [conn10] build index dicomdb.dicom { files_id: 1.0 } Sun Nov 24 14:51:20.061 [conn10] build index done. scanned 0 total records. 0.027 secs Sun Nov 24 14:51:20.565 [conn3626] no current chunk manager found for this shard, will initialize Sun Nov 24 14:51:22.022 [conn10] build index dicomdb.fs.files { _id: 1 } Sun Nov 24 14:51:22.024 [conn10] build index done. scanned 0 total records. 0.001 secs Sun Nov 24 14:51:22.025 [conn10] info: creating collection dicomdb.fs.files on add index Sun Nov 24 14:51:22.664 [conn3626] no current chunk manager found for this shard, will initialize Sun Nov 24 14:51:51.535 [conn3640] build index dicomdb.fs.files { filename: 1, uploadDate: 1 } Sun Nov 24 14:51:51.538 [conn3640] build index done. scanned 0 total records. 0.002 secs Sun Nov 24 14:51:51.564 [conn3640] build index dicomdb.fs.chunks { files_id: 1, n: 1 } Sun Nov 24 14:51:51.584 [conn3640] build index done. scanned 0 total records. 0.02 secs Sun Nov 24 14:51:51.659 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.659 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:51:51.660 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.660 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:51:51.777 [conn7] request split points lookup for chunk dicomdb.dicom { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.777 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:51:51.799 [conn7] request split points lookup for chunk dicomdb.dicom { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.799 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:51:51.801 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.801 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.801 [conn7] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:51:51.802 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d70cf23a90f681ad59') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:51.818 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.818 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.819 [conn10] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:51:51.820 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d70cf23a90f681ad59') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:51.867 [conn10] request split points lookup for chunk dicomdb.dicom { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.867 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:51.867 [conn10] warning: chunk is larger than 1024 bytes because of key { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:51:51.867 [conn10] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d70cf23a90f681ad59') } ], shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:51.953 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258d718c2315fef764aaf Sun Nov 24 14:51:51.955 [conn7] splitChunk accepted at version 1|0||529258b4c868398bb45e3755 Sun Nov 24 14:51:52.024 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:51:52-529258d818c2315fef764ab1", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322712024), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: MinKey }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:51:52.024 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258d718c2315fef764ab0 Sun Nov 24 14:51:52.026 [conn10] splitChunk accepted at version 1|0||529258b8c868398bb45e3756 Sun Nov 24 14:51:52.448 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:51:52-529258d818c2315fef764ab2", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322712448), what: "split", ns: "dicomdb.dicom", details: { before: { min: { files_id: MinKey }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('529258b8c868398bb45e3756') }, right: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('529258b8c868398bb45e3756') } } } Sun Nov 24 14:51:52.763 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:51:52.763 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d70cf23a90f681ad59') } ], shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:144 reslen:37 961ms Sun Nov 24 14:51:52.774 [conn7] request split points lookup for chunk dicomdb.fs.files { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:52.774 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.files { : MinKey } -->> { : MaxKey } Sun Nov 24 14:51:52.775 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('529258d70cf23a90f681ad59') } ], shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:52.869 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:51:52.869 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: MinKey }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d70cf23a90f681ad59') } ], shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:137 reslen:37 1002ms Sun Nov 24 14:51:52.897 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:52.897 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:52.899 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d80cf23a90f681ad65') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:52.909 [conn7] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258d818c2315fef764ab3 Sun Nov 24 14:51:52.910 [conn7] splitChunk accepted at version 1|0||529258bac868398bb45e3757 Sun Nov 24 14:51:52.975 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:51:52-529258d818c2315fef764ab5", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322712975), what: "split", ns: "dicomdb.fs.files", details: { before: { min: { _id: MinKey }, max: { _id: MaxKey }, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('529258bac868398bb45e3757') }, right: { min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('529258bac868398bb45e3757') } } } Sun Nov 24 14:51:53.046 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258d818c2315fef764ab4 Sun Nov 24 14:51:53.048 [conn10] splitChunk accepted at version 1|2||529258b4c868398bb45e3755 Sun Nov 24 14:51:53.572 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:51:53-529258d918c2315fef764ab6", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322713572), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, lastmod: Timestamp 1000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|4, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:51:53.572 [conn7] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:51:53.572 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('529258d70cf23a90f681ad59') } ], shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:109 reslen:37 797ms Sun Nov 24 14:51:53.606 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.606 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.608 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d90cf23a90f681ad68') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:53.662 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.662 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.663 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d90cf23a90f681ad6b') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:53.689 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.689 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.690 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d90cf23a90f681ad6e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:53.715 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.715 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.717 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d90cf23a90f681ad71') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:53.743 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.743 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:53.745 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d90cf23a90f681ad74') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:53.757 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:51:53.757 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d80cf23a90f681ad65') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:46 reslen:37 858ms Sun Nov 24 14:51:54.042 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258d918c2315fef764ab7 Sun Nov 24 14:51:54.042 [conn7] warning: aborted split because chunk boundaries are outdated (likely a split occurred): { files_id: ObjectId('529258d70cf23a90f681ad59') }->{ files_id: MaxKey } is now { files_id: ObjectId('529258d70cf23a90f681ad59') }->{ files_id: ObjectId('529258d80cf23a90f681ad65') } Sun Nov 24 14:51:54.287 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:195 110ms Sun Nov 24 14:51:54.588 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:384054 192ms Sun Nov 24 14:51:54.820 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:51:54.821 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258d90cf23a90f681ad74') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:247 1076ms Sun Nov 24 14:51:55.016 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:215053 108ms Sun Nov 24 14:51:55.254 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:247 127ms Sun Nov 24 14:51:55.255 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:255158 127ms Sun Nov 24 14:51:55.477 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:213975 107ms Sun Nov 24 14:51:55.979 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:217 178ms Sun Nov 24 14:51:55.980 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:357106 179ms Sun Nov 24 14:51:56.293 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:402272 201ms Sun Nov 24 14:51:56.739 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:659065 329ms Sun Nov 24 14:51:57.184 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:648454 324ms Sun Nov 24 14:51:57.641 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:681931 341ms Sun Nov 24 14:51:57.701 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.2, filling with zeroes... Sun Nov 24 14:51:58.753 [conn10] request split points lookup for chunk dicomdb.dicom { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:58.759 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:58.760 [conn10] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258de0cf23a90f681aff5') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:51:58.766 [conn7] request split points lookup for chunk dicomdb.dicom { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:58.767 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:51:58.768 [conn7] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258de0cf23a90f681aff5') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:01.325 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.2, size: 256MB, took 3.623 secs Sun Nov 24 14:52:01.686 [conn10] warning: distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.dicom", state: 0, ts: ObjectId('529258d718c2315fef764ab0') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn10:1345868337", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322718760), why: "split-{ files_id: ObjectId('529258d70cf23a90f681ad59') }", ts: ObjectId('529258de18c2315fef764ab8') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322721000|1, connectionId: 29, waited: 53, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322714000|2, connectionId: 62, waited: 21, err: null, ok: 1.0 } Sun Nov 24 14:52:01.686 [conn7] warning: distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.dicom", state: 0, ts: ObjectId('529258d718c2315fef764ab0') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322718768), why: "split-{ files_id: ObjectId('529258d70cf23a90f681ad59') }", ts: ObjectId('529258de18c2315fef764ab9') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322713000|4, connectionId: 28, waited: 52, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322721000|2, connectionId: 61, waited: 21, err: null, ok: 1.0 } Sun Nov 24 14:52:01.687 [conn10] lock update lost, lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:52:01.687 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:52:01.687 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258de0cf23a90f681aff5') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:376 2927ms Sun Nov 24 14:52:01.689 [conn7] lock update won, completing lock propagation for 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:52:01.956 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:367934 184ms Sun Nov 24 14:52:02.286 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:427876 214ms Sun Nov 24 14:52:02.286 [conn7] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258de18c2315fef764ab9 Sun Nov 24 14:52:02.288 [conn7] splitChunk accepted at version 1|2||529258b8c868398bb45e3756 Sun Nov 24 14:52:02.350 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:52:02.351 [conn10] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:52:02.354 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:02.355 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:02.357 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e20cf23a90f681b063') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:02.568 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:02-529258e218c2315fef764abc", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322722568), what: "split", ns: "dicomdb.dicom", details: { before: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, lastmod: Timestamp 1000|3, lastmodEpoch: ObjectId('529258b8c868398bb45e3756') }, right: { min: { files_id: ObjectId('529258de0cf23a90f681aff5') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|4, lastmodEpoch: ObjectId('529258b8c868398bb45e3756') } } } Sun Nov 24 14:52:02.568 [conn3641] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:52:02.568 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:52:02.568 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e20cf23a90f681b063') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:353 210ms Sun Nov 24 14:52:02.940 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:529336 264ms Sun Nov 24 14:52:02.992 [conn3641] request split points lookup for chunk dicomdb.dicom { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:52:02.992 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.dicom { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:52:02.993 [conn3641] received splitChunk request: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e20cf23a90f681b0a4') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:02.997 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258e218c2315fef764aba Sun Nov 24 14:52:02.997 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:02-529258e218c2315fef764abd", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322722997), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:52:03.172 [conn7] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:03.173 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258de0cf23a90f681aff5') } ], shardId: "dicomdb.dicom-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:132 reslen:37 4404ms Sun Nov 24 14:52:03.173 [conn10] moveChunk request accepted at version 1|4||529258b4c868398bb45e3755 Sun Nov 24 14:52:03.269 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:428576 214ms Sun Nov 24 14:52:03.272 [conn10] moveChunk number of documents: 0 Sun Nov 24 14:52:03.275 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.279 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.285 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.316 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:03.319 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:03.319 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:03.320 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:03.322 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e30cf23a90f681b0d8') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:03.322 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e30cf23a90f681b0d8') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:03.329 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.348 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.382 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.448 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.578 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:03.836 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:04.331 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:961224 481ms Sun Nov 24 14:52:04.346 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.346 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.346 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.347 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.349 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e30cf23a90f681b145') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:04.349 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e30cf23a90f681b145') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:04.350 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:04.362 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.363 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.365 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e40cf23a90f681b14e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:04.368 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.368 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.371 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e40cf23a90f681b14e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:04.741 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:587631 294ms Sun Nov 24 14:52:04.841 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.842 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.843 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.844 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e40cf23a90f681b1c2') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:04.844 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:04.846 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e40cf23a90f681b1c2') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:05.159 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:595775 298ms Sun Nov 24 14:52:05.162 [conn3630] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:31 reslen:51 203ms Sun Nov 24 14:52:05.376 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:05.880 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1205024 603ms Sun Nov 24 14:52:05.883 [conn10] moveChunk setting version to: 2|0||529258b4c868398bb45e3755 Sun Nov 24 14:52:05.884 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:51 reslen:51 610ms Sun Nov 24 14:52:05.884 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:262523 88018ms Sun Nov 24 14:52:05.886 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:52:05.886 [conn10] moveChunk updating self version to: 2|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258d80cf23a90f681ad65') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:52:05.888 [conn3642] waiting till out of critical section Sun Nov 24 14:52:07.693 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:07-529258e718c2315fef764abe", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322727693), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:52:07.693 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 1000|4, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:106 reslen:325 1805ms Sun Nov 24 14:52:07.789 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:07.789 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:07.792 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e70cf23a90f681b21d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:07.798 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:07.801 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:07.803 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e70cf23a90f681b21d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:08.156 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:08.402 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:268566 267ms Sun Nov 24 14:52:08.403 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:08.403 [conn10] forking for cleanup of chunk data Sun Nov 24 14:52:08.403 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:08.403 [cleanupOldData-529258e818c2315fef764abf] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') }, # cursors remaining: 0 Sun Nov 24 14:52:08.405 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:08.425 [cleanupOldData-529258e818c2315fef764abf] waiting to remove documents for dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:08.425 [cleanupOldData-529258e818c2315fef764abf] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:08.425 [cleanupOldData-529258e818c2315fef764abf] moveChunk deleted 0 documents for dicomdb.fs.chunks from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:08.446 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:08.446 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:08.447 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:08.447 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:08.449 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e80cf23a90f681b28f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:08.449 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e80cf23a90f681b28f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:08.449 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:52:08.450 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:52:08.450 [conn7] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:52:08.722 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:08.722 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:08-529258e818c2315fef764ac0", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322728722), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, step1 of 6: 1, step2 of 6: 821, step3 of 6: 99, step4 of 6: 2103, step5 of 6: 3026, step6 of 6: 0 } } Sun Nov 24 14:52:08.783 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:533349 266ms Sun Nov 24 14:52:08.863 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:219 r:209 w:43 reslen:37 6513ms Sun Nov 24 14:52:08.865 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:52:08.866 [conn10] received moveChunk request: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:52:08.888 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:08.890 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:08.892 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e80cf23a90f681b303') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:09.041 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258e818c2315fef764ac1 Sun Nov 24 14:52:09.041 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258e818c2315fef764ac2 Sun Nov 24 14:52:09.041 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:09-529258e918c2315fef764ac3", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322729041), what: "moveChunk.start", ns: "dicomdb.dicom", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:52:09.042 [conn7] splitChunk accepted at version 2|1||529258b4c868398bb45e3755 Sun Nov 24 14:52:09.266 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:720998 361ms Sun Nov 24 14:52:09.275 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:09.275 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:09.278 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e80cf23a90f681b30b') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:09.294 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:09.295 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : MaxKey } Sun Nov 24 14:52:09.296 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e80cf23a90f681b30b') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:09.391 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:09-529258e918c2315fef764ac4", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322729391), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, lastmod: Timestamp 1000|4, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:52:09.392 [conn10] moveChunk request accepted at version 1|4||529258b8c868398bb45e3756 Sun Nov 24 14:52:09.595 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:431182 216ms Sun Nov 24 14:52:09.596 [conn10] moveChunk number of documents: 0 Sun Nov 24 14:52:09.599 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:09.603 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:09.609 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:09.619 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:09.637 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:09.671 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:09.672 [conn10] moveChunk setting version to: 2|0||529258b8c868398bb45e3756 Sun Nov 24 14:52:09.675 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:52:09.675 [conn10] moveChunk updating self version to: 2|1||529258b8c868398bb45e3756 through { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258de0cf23a90f681aff5') } for collection 'dicomdb.dicom' Sun Nov 24 14:52:09.676 [conn3642] waiting till out of critical section Sun Nov 24 14:52:09.920 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:09-529258e918c2315fef764ac5", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322729920), what: "moveChunk.commit", ns: "dicomdb.dicom", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:52:09.920 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.dicom", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 1000|4, versionEpoch: ObjectId('529258b8c868398bb45e3756'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:120 reslen:317 244ms Sun Nov 24 14:52:10.061 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:10.061 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258e80cf23a90f681b303') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:154 reslen:37 1169ms Sun Nov 24 14:52:10.062 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:10.154 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:10.154 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:254103 127ms Sun Nov 24 14:52:10.154 [conn10] forking for cleanup of chunk data Sun Nov 24 14:52:10.154 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:10.154 [cleanupOldData-529258ea18c2315fef764ac6] (start) waiting to cleanup dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') }, # cursors remaining: 0 Sun Nov 24 14:52:10.154 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:10.163 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : ObjectId('529258e80cf23a90f681b303') } Sun Nov 24 14:52:10.166 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258d80cf23a90f681ad65') } -->> { : ObjectId('529258e80cf23a90f681b303') } Sun Nov 24 14:52:10.166 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258dc0cf23a90f681aef9') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:10.176 [cleanupOldData-529258ea18c2315fef764ac6] waiting to remove documents for dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:10.176 [cleanupOldData-529258ea18c2315fef764ac6] moveChunk starting delete for: dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:10.176 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:10.176 [cleanupOldData-529258ea18c2315fef764ac6] moveChunk deleted 0 documents for dicomdb.dicom from { files_id: MinKey } -> { files_id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:10.383 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:10.384 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:10-529258ea18c2315fef764ac8", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322730383), what: "moveChunk.from", ns: "dicomdb.dicom", details: { min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, step1 of 6: 1, step2 of 6: 526, step3 of 6: 203, step4 of 6: 74, step5 of 6: 482, step6 of 6: 0 } } Sun Nov 24 14:52:10.491 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:444181 222ms Sun Nov 24 14:52:10.558 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { files_id: MinKey }, max: { files_id: ObjectId('529258d70cf23a90f681ad59') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:224 r:230 w:35 reslen:37 1693ms Sun Nov 24 14:52:10.558 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258ea18c2315fef764ac7 Sun Nov 24 14:52:10.560 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:52:10.561 [conn7] splitChunk accepted at version 2|3||529258b4c868398bb45e3755 Sun Nov 24 14:52:10.561 [conn10] received moveChunk request: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:52:10.699 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:10-529258ea18c2315fef764aca", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322730699), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, lastmod: Timestamp 2000|4, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, lastmod: Timestamp 2000|5, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:52:10.805 [conn10] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258ea18c2315fef764ac9 Sun Nov 24 14:52:10.805 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:10-529258ea18c2315fef764acb", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322730805), what: "moveChunk.start", ns: "dicomdb.fs.files", details: { min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:52:10.930 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:87 322ms Sun Nov 24 14:52:10.946 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:11.016 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:11.016 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258dc0cf23a90f681aef9') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:116 reslen:37 850ms Sun Nov 24 14:52:11.017 [conn10] moveChunk request accepted at version 1|2||529258bac868398bb45e3757 Sun Nov 24 14:52:11.017 [conn10] moveChunk number of documents: 0 Sun Nov 24 14:52:11.021 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:11.021 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.3, filling with zeroes... Sun Nov 24 14:52:11.025 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:11.032 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { _id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:11.036 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:11.041 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { _id: 1.0 }, state: "catchup", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:11.058 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { _id: 1.0 }, state: "steady", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:11.247 [conn10] moveChunk setting version to: 2|0||529258bac868398bb45e3757 Sun Nov 24 14:52:11.247 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:11.248 [conn3630] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:40 reslen:51 191ms Sun Nov 24 14:52:11.249 [conn3642] waiting till out of critical section Sun Nov 24 14:52:11.250 [conn7] command admin.$cmd command: { splitVector: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) r:1025 reslen:69 210ms Sun Nov 24 14:52:11.250 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, shardKeyPattern: { _id: 1.0 }, state: "done", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:52:11.250 [conn10] moveChunk updating self version to: 2|1||529258bac868398bb45e3757 through { _id: ObjectId('529258d70cf23a90f681ad59') } -> { _id: MaxKey } for collection 'dicomdb.fs.files' Sun Nov 24 14:52:11.367 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:11-529258eb18c2315fef764acc", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322731367), what: "moveChunk.commit", ns: "dicomdb.fs.files", details: { min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, from: "dicom1", to: "dicom10" } } Sun Nov 24 14:52:11.367 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.files", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 1000|2, versionEpoch: ObjectId('529258bac868398bb45e3757'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:78 reslen:323 118ms Sun Nov 24 14:52:12.016 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:12.017 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:12.639 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:247 567ms Sun Nov 24 14:52:12.639 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:233 567ms Sun Nov 24 14:52:13.209 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:902385 451ms Sun Nov 24 14:52:13.234 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:13.239 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:13.931 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1204525 602ms Sun Nov 24 14:52:14.566 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1041716 520ms Sun Nov 24 14:52:14.588 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:14.589 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:14.590 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:14.590 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:14.591 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258ee0cf23a90f681b56f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:14.593 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258ee0cf23a90f681b56f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:19.337 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.3, size: 512MB, took 8.316 secs Sun Nov 24 14:52:20.050 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:20.050 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:20.050 [conn10] forking for cleanup of chunk data Sun Nov 24 14:52:20.050 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:20.050 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:20.051 [cleanupOldData-529258f418c2315fef764acf] (start) waiting to cleanup dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529258d70cf23a90f681ad59') }, # cursors remaining: 0 Sun Nov 24 14:52:20.051 [conn3641] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:52:20.051 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:52:20.051 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258ee0cf23a90f681b56f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:380 5458ms Sun Nov 24 14:52:20.071 [cleanupOldData-529258f418c2315fef764acf] waiting to remove documents for dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:20.072 [cleanupOldData-529258f418c2315fef764acf] moveChunk starting delete for: dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:20.073 [cleanupOldData-529258f418c2315fef764acf] moveChunk deleted 0 documents for dicomdb.fs.files from { _id: MinKey } -> { _id: ObjectId('529258d70cf23a90f681ad59') } Sun Nov 24 14:52:20.197 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:208382 104ms Sun Nov 24 14:52:20.508 [conn10] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:20.508 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:20-529258f418c2315fef764ad0", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322740508), what: "moveChunk.from", ns: "dicomdb.fs.files", details: { min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, step1 of 6: 1, step2 of 6: 456, step3 of 6: 1, step4 of 6: 39, step5 of 6: 8992, step6 of 6: 0 } } Sun Nov 24 14:52:20.508 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258ee18c2315fef764acd Sun Nov 24 14:52:20.509 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:52:20.509 [conn7] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:52:20.510 [conn7] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:52:20.512 [conn7] splitChunk accepted at version 2|5||529258b4c868398bb45e3755 Sun Nov 24 14:52:20.543 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:461731 231ms Sun Nov 24 14:52:20.636 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:20.656 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:20.657 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f40cf23a90f681b5e6') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:20.867 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:414425 207ms Sun Nov 24 14:52:20.930 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom10/node40.clus.cci.emory.edu:27010", fromShard: "dicom1", toShard: "dicom10", min: { _id: MinKey }, max: { _id: ObjectId('529258d70cf23a90f681ad59') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_MinKey", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:40 r:153 w:42 reslen:37 10370ms Sun Nov 24 14:52:20.930 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:20-529258f418c2315fef764ad1", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322740930), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, lastmod: Timestamp 2000|6, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, lastmod: Timestamp 2000|7, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:52:21.271 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:577707 289ms Sun Nov 24 14:52:21.351 [LockPinger] trying to delete 1 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:52:21.384 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:21.385 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258e80cf23a90f681b303') } -->> { : MaxKey } Sun Nov 24 14:52:21.387 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f50cf23a90f681b659') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:21.888 [LockPinger] handled late remove of old distributed lock with ts 529258de18c2315fef764ab8 Sun Nov 24 14:52:21.888 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:21.888 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258ee0cf23a90f681b56f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:154 reslen:37 7297ms Sun Nov 24 14:52:21.905 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:22.065 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258f518c2315fef764ad2 Sun Nov 24 14:52:22.066 [conn10] warning: aborted split because chunk boundaries are outdated (likely a split occurred): { files_id: ObjectId('529258e80cf23a90f681b303') }->{ files_id: MaxKey } is now { files_id: ObjectId('529258e80cf23a90f681b303') }->{ files_id: ObjectId('529258ee0cf23a90f681b56f') } Sun Nov 24 14:52:22.206 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:22.206 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f50cf23a90f681b659') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:247 819ms Sun Nov 24 14:52:22.455 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:505841 253ms Sun Nov 24 14:52:22.510 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:22.517 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:22.972 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:788905 394ms Sun Nov 24 14:52:22.975 [conn10] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:102 reslen:3137 334ms Sun Nov 24 14:52:23.009 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:52:23.010 [conn10] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:52:23.082 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:23.082 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:23.507 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:831173 416ms Sun Nov 24 14:52:23.732 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258f718c2315fef764ad3 Sun Nov 24 14:52:23.733 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:23-529258f718c2315fef764ad4", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322743733), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:52:23.805 [conn10] moveChunk request accepted at version 2|7||529258b4c868398bb45e3755 Sun Nov 24 14:52:24.015 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:766525 383ms Sun Nov 24 14:52:24.017 [conn10] moveChunk number of documents: 4 Sun Nov 24 14:52:24.021 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.024 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.029 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.038 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.055 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.070 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.071 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.073 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f80cf23a90f681b79f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:24.075 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.075 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.077 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f80cf23a90f681b79f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:24.089 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.156 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.285 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.542 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:24.623 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:967127 484ms Sun Nov 24 14:52:24.729 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.730 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.732 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.732 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f80cf23a90f681b80e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:24.732 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:24.735 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f80cf23a90f681b80e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:25.055 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:25.397 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1312620 656ms Sun Nov 24 14:52:26.022 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.023 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.024 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fa0cf23a90f681b886') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:26.025 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.025 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.027 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fa0cf23a90f681b886') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:26.035 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.036 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.037 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fa0cf23a90f681b886') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:26.038 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.039 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:26.042 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fa0cf23a90f681b886') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:26.193 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 4, clonedBytes: 799490, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:26.340 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:536052 268ms Sun Nov 24 14:52:26.342 [conn10] moveChunk setting version to: 3|0||529258b4c868398bb45e3755 Sun Nov 24 14:52:26.342 [conn3638] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:154 reslen:51 275ms Sun Nov 24 14:52:26.432 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 4, clonedBytes: 799490, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:52:26.432 [conn10] moveChunk updating self version to: 3|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529258d80cf23a90f681ad65') } -> { files_id: ObjectId('529258dc0cf23a90f681aef9') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:52:26.514 [conn3642] waiting till out of critical section Sun Nov 24 14:52:28.704 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:28-529258fc18c2315fef764ad5", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322748704), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:52:28.705 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 2000|7, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:102 reslen:325 2190ms Sun Nov 24 14:52:28.780 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:28.781 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:28.783 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fc0cf23a90f681b8a6') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:28.786 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:28.786 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:28.788 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fc0cf23a90f681b8a6') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:29.091 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:29.146 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:534751 267ms Sun Nov 24 14:52:29.146 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:29.146 [conn10] forking for cleanup of chunk data Sun Nov 24 14:52:29.146 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:29.147 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:29.147 [cleanupOldData-529258fd18c2315fef764ad6] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258d80cf23a90f681ad65') }, # cursors remaining: 0 Sun Nov 24 14:52:29.168 [cleanupOldData-529258fd18c2315fef764ad6] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258d80cf23a90f681ad65') } Sun Nov 24 14:52:29.168 [cleanupOldData-529258fd18c2315fef764ad6] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258d80cf23a90f681ad65') } Sun Nov 24 14:52:29.172 [cleanupOldData-529258fd18c2315fef764ad6] moveChunk deleted 4 documents for dicomdb.fs.chunks from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258d80cf23a90f681ad65') } Sun Nov 24 14:52:29.235 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:29.237 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:29.253 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:29.253 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:29.255 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fd0cf23a90f681b917') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:29.255 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fd0cf23a90f681b917') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:29.513 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:29.514 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:29-529258fd18c2315fef764ad7", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322749514), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, step1 of 6: 1, step2 of 6: 795, step3 of 6: 213, step4 of 6: 2174, step5 of 6: 2953, step6 of 6: 0 } } Sun Nov 24 14:52:29.622 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258d80cf23a90f681ad65') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:177 r:278 w:49 reslen:37 6612ms Sun Nov 24 14:52:29.623 [conn10] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:52:29.625 [conn10] received moveChunk request: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:52:29.834 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258fd18c2315fef764ad8 Sun Nov 24 14:52:29.834 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:29-529258fd18c2315fef764ad9", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322749834), what: "moveChunk.start", ns: "dicomdb.dicom", details: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:52:30.020 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:723510 362ms Sun Nov 24 14:52:30.075 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:30.076 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:30.078 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fe0cf23a90f681b98f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:30.078 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:30.079 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:30.083 [conn10] moveChunk request accepted at version 2|1||529258b8c868398bb45e3756 Sun Nov 24 14:52:30.083 [conn10] moveChunk number of documents: 174 Sun Nov 24 14:52:30.084 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fe0cf23a90f681b98f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:30.087 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:30.090 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:30.095 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:30.104 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 47, clonedBytes: 141988, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:30.121 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 174, clonedBytes: 500974, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:30.154 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 174, clonedBytes: 500974, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:30.219 [conn10] moveChunk data transfer progress: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 174, clonedBytes: 500974, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:52:30.219 [conn10] moveChunk setting version to: 3|0||529258b8c868398bb45e3756 Sun Nov 24 14:52:30.228 [conn10] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 174, clonedBytes: 500974, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:52:30.228 [conn10] moveChunk updating self version to: 3|1||529258b8c868398bb45e3756 through { files_id: ObjectId('529258de0cf23a90f681aff5') } -> { files_id: MaxKey } for collection 'dicomdb.dicom' Sun Nov 24 14:52:30.619 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:30-529258fe18c2315fef764adc", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322750619), what: "moveChunk.commit", ns: "dicomdb.dicom", details: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:52:30.619 [conn3643] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529258f718c2315fef764ad3') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn3643:179229628", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322750078), why: "split-{ files_id: ObjectId('529258ee0cf23a90f681b56f') }", ts: ObjectId('529258fe18c2315fef764ada') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322750000|3, connectionId: 42, waited: 46, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322749000|4, connectionId: 75, waited: 21, err: null, ok: 1.0 } Sun Nov 24 14:52:30.619 [conn7] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529258f718c2315fef764ad3') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322750085), why: "split-{ files_id: ObjectId('529258ee0cf23a90f681b56f') }", ts: ObjectId('529258fe18c2315fef764adb') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322743000|2, connectionId: 29, waited: 45, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322750000|1, connectionId: 62, waited: 21, err: null, ok: 1.0 } Sun Nov 24 14:52:30.620 [conn3643] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:52:30.620 [conn3643] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:52:30.621 [conn3643] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fe0cf23a90f681b98f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:380 542ms Sun Nov 24 14:52:30.623 [conn7] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:52:30.636 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:30.636 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : MaxKey } Sun Nov 24 14:52:30.638 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fe0cf23a90f681b98f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:30.727 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:30.728 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258fe18c2315fef764adb Sun Nov 24 14:52:30.728 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:30.728 [conn10] forking for cleanup of chunk data Sun Nov 24 14:52:30.728 [conn10] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:52:30.728 [cleanupOldData-529258fe18c2315fef764add] (start) waiting to cleanup dicomdb.dicom from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258de0cf23a90f681aff5') }, # cursors remaining: 0 Sun Nov 24 14:52:30.728 [conn10] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:52:30.730 [conn7] splitChunk accepted at version 3|1||529258b4c868398bb45e3755 Sun Nov 24 14:52:30.749 [cleanupOldData-529258fe18c2315fef764add] waiting to remove documents for dicomdb.dicom from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258de0cf23a90f681aff5') } Sun Nov 24 14:52:30.750 [cleanupOldData-529258fe18c2315fef764add] moveChunk starting delete for: dicomdb.dicom from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258de0cf23a90f681aff5') } Sun Nov 24 14:52:30.801 [conn10] distributed lock 'dicomdb.dicom/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:30.801 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:30-529258fe18c2315fef764ade", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322750801), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, lastmod: Timestamp 2000|7, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:52:30.801 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:30-529258fe18c2315fef764adf", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322750801), what: "moveChunk.from", ns: "dicomdb.dicom", details: { min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, step1 of 6: 1, step2 of 6: 458, step3 of 6: 0, step4 of 6: 134, step5 of 6: 509, step6 of 6: 0 } } Sun Nov 24 14:52:30.981 [conn10] command admin.$cmd command: { moveChunk: "dicomdb.dicom", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { files_id: ObjectId('529258d70cf23a90f681ad59') }, max: { files_id: ObjectId('529258de0cf23a90f681aff5') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.dicom-files_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:118 r:532 w:25 reslen:37 1357ms Sun Nov 24 14:52:31.011 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:466564 233ms Sun Nov 24 14:52:31.040 [cleanupOldData-529258fe18c2315fef764add] moveChunk deleted 174 documents for dicomdb.dicom from { files_id: ObjectId('529258d70cf23a90f681ad59') } -> { files_id: ObjectId('529258de0cf23a90f681aff5') } Sun Nov 24 14:52:31.084 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.4, filling with zeroes... Sun Nov 24 14:52:31.089 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:31.089 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258fe0cf23a90f681b98f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:130 reslen:37 1004ms Sun Nov 24 14:52:31.102 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : ObjectId('529258fe0cf23a90f681b98f') } Sun Nov 24 14:52:31.102 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258ee0cf23a90f681b56f') } -->> { : ObjectId('529258fe0cf23a90f681b98f') } Sun Nov 24 14:52:31.103 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f60cf23a90f681b682') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:31.104 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:31.530 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:807792 404ms Sun Nov 24 14:52:31.976 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:662960 331ms Sun Nov 24 14:52:32.531 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:880368 440ms Sun Nov 24 14:52:33.126 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:33.807 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:258 596ms Sun Nov 24 14:52:36.490 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:181 2566ms Sun Nov 24 14:52:36.547 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:36.866 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:523621 262ms Sun Nov 24 14:52:37.563 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1162343 581ms Sun Nov 24 14:52:37.639 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:37.960 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:564897 282ms Sun Nov 24 14:52:38.237 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:52:38.467 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:781938 391ms Sun Nov 24 14:52:38.549 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:38.550 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:38.552 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259060cf23a90f681bbc3') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:46.496 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.4, size: 1024MB, took 15.411 secs Sun Nov 24 14:52:47.231 [conn10] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:52:47.232 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:52:47.232 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259060cf23a90f681bbc3') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:380 8679ms Sun Nov 24 14:52:47.447 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:300651 150ms Sun Nov 24 14:52:47.513 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529258ff18c2315fef764ae0 Sun Nov 24 14:52:47.515 [conn7] splitChunk accepted at version 3|3||529258b4c868398bb45e3755 Sun Nov 24 14:52:47.559 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:52:47.560 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:52:47.562 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:47-5292590f18c2315fef764ae2", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322767562), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, step1 of 6: 0, note: "aborted" } } Sun Nov 24 14:52:47.724 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:453 164ms Sun Nov 24 14:52:47.724 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:47-5292590f18c2315fef764ae3", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322767724), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, lastmod: Timestamp 3000|2, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, lastmod: Timestamp 3000|4, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, lastmod: Timestamp 3000|5, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:52:47.834 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:539304 270ms Sun Nov 24 14:52:47.907 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:47.908 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:47.910 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292590f0cf23a90f681bc36') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:48.436 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:48.436 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529258f60cf23a90f681b682') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:141 reslen:37 17333ms Sun Nov 24 14:52:48.560 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:571308 285ms Sun Nov 24 14:52:48.577 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:48.577 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:48.579 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:48.580 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:48.580 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259100cf23a90f681bc87') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:48.582 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259100cf23a90f681bc87') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:48.647 [conn7] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529258ff18c2315fef764ae0') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322768581), why: "split-{ files_id: ObjectId('529258fe0cf23a90f681b98f') }", ts: ObjectId('5292591018c2315fef764ae4') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322768000|4, connectionId: 29, waited: 32, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322768000|5, connectionId: 62, waited: 19, err: null, ok: 1.0 } Sun Nov 24 14:52:48.647 [conn10] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529258ff18c2315fef764ae0') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn10:1345868337", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322768583), why: "split-{ files_id: ObjectId('529258fe0cf23a90f681b98f') }", ts: ObjectId('5292591018c2315fef764ae5') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322768000|5, connectionId: 42, waited: 32, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322749000|4, connectionId: 75, waited: 19, err: null, ok: 1.0 } Sun Nov 24 14:52:48.648 [conn7] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:52:48.649 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:52:48.650 [conn10] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:52:48.753 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292591018c2315fef764ae5 Sun Nov 24 14:52:48.755 [conn10] splitChunk accepted at version 3|5||529258b4c868398bb45e3755 Sun Nov 24 14:52:48.791 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:238407 119ms Sun Nov 24 14:52:48.859 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:52:48-5292591018c2315fef764ae6", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322768859), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, lastmod: Timestamp 3000|6, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|7, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:52:49.205 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:588894 295ms Sun Nov 24 14:52:49.300 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:49.301 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529258fe0cf23a90f681b98f') } -->> { : MaxKey } Sun Nov 24 14:52:49.303 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259110cf23a90f681bcfe') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:49.353 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:49.353 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259100cf23a90f681bc87') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:175 reslen:37 771ms Sun Nov 24 14:52:49.584 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:3224 225ms Sun Nov 24 14:52:49.593 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:49.855 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292591118c2315fef764ae7 Sun Nov 24 14:52:49.857 [conn7] warning: aborted split because chunk boundaries are outdated (likely a split occurred): { files_id: ObjectId('529258fe0cf23a90f681b98f') }->{ files_id: MaxKey } is now { files_id: ObjectId('529258fe0cf23a90f681b98f') }->{ files_id: ObjectId('529259100cf23a90f681bc87') } Sun Nov 24 14:52:49.955 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:514145 257ms Sun Nov 24 14:52:50.100 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:52:50.100 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259110cf23a90f681bcfe') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:247 797ms Sun Nov 24 14:52:50.538 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:932205 466ms Sun Nov 24 14:52:50.570 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:50.572 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:51.392 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1471260 736ms Sun Nov 24 14:52:52.120 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1227267 614ms Sun Nov 24 14:52:52.147 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:52.147 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:52.538 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:612575 306ms Sun Nov 24 14:52:53.261 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:53.263 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:54.699 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:284 603ms Sun Nov 24 14:52:54.699 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:1205951 603ms Sun Nov 24 14:52:54.793 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:54.796 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:55.307 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:985938 493ms Sun Nov 24 14:52:56.164 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:273 739ms Sun Nov 24 14:52:56.164 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:1478875 739ms Sun Nov 24 14:52:56.735 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:917618 459ms Sun Nov 24 14:52:56.791 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:56.791 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:56.793 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259180cf23a90f681c05a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:56.793 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:56.794 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:52:56.795 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259180cf23a90f681c05a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:52:58.762 [LockPinger] trying to delete 2 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:52:58.763 [conn10] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:52:58.763 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:52:58.763 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259180cf23a90f681c05a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:380 1967ms Sun Nov 24 14:52:59.142 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:234 290ms Sun Nov 24 14:52:59.792 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1076985 538ms Sun Nov 24 14:53:00.244 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:680769 340ms Sun Nov 24 14:53:00.299 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:00.360 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:00.362 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292591c0cf23a90f681c10f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:02.178 [LockPinger] handled late remove of old distributed lock with ts 529258fe18c2315fef764ada Sun Nov 24 14:53:02.178 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292591818c2315fef764ae8 Sun Nov 24 14:53:02.181 [conn7] splitChunk accepted at version 3|7||529258b4c868398bb45e3755 Sun Nov 24 14:53:02.213 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:53:02.215 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:53:04.391 [LockPinger] handled late remove of old distributed lock with ts 5292591018c2315fef764ae4 Sun Nov 24 14:53:04.391 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:53:04-5292592018c2315fef764aea", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322784391), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|7, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, lastmod: Timestamp 3000|8, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|9, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:53:04.393 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:53:04-5292592018c2315fef764aeb", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322784393), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, step1 of 6: 1, note: "aborted" } } Sun Nov 24 14:53:04.394 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292591c0cf23a90f681c10f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:380 4031ms Sun Nov 24 14:53:04.872 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:497944 249ms Sun Nov 24 14:53:05.253 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:538111 269ms Sun Nov 24 14:53:05.361 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:05.362 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:05.363 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259210cf23a90f681c1ba') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:05.691 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:229 324ms Sun Nov 24 14:53:06.172 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:738335 369ms Sun Nov 24 14:53:06.809 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:561222 281ms Sun Nov 24 14:53:07.072 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:308518 154ms Sun Nov 24 14:53:07.183 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:07.184 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:07.186 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259230cf23a90f681c277') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:07.462 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:187 272ms Sun Nov 24 14:53:10.562 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:453 8348ms Sun Nov 24 14:53:11.757 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:7215793 3608ms Sun Nov 24 14:53:12.235 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:715512 358ms Sun Nov 24 14:53:12.807 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:457124 228ms Sun Nov 24 14:53:12.849 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:12.850 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259100cf23a90f681bc87') } -->> { : MaxKey } Sun Nov 24 14:53:12.852 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259280cf23a90f681c300') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:13.326 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:808704 404ms Sun Nov 24 14:53:13.388 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:13.507 [DataFileSync] flushing mmaps took 23555ms for 9 files Sun Nov 24 14:53:13.593 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:313311 157ms Sun Nov 24 14:53:14.013 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:612766 306ms Sun Nov 24 14:53:14.095 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:53:14.096 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259180cf23a90f681c05a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:170 reslen:37 17302ms Sun Nov 24 14:53:14.399 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:540832 270ms Sun Nov 24 14:53:14.903 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:767379 383ms Sun Nov 24 14:53:14.965 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:14.966 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:15.738 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:574699 287ms Sun Nov 24 14:53:16.237 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:742593 371ms Sun Nov 24 14:53:16.307 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:16.309 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:16.671 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:247 317ms Sun Nov 24 14:53:16.671 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:256 317ms Sun Nov 24 14:53:17.301 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1016776 508ms Sun Nov 24 14:53:17.895 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:934408 467ms Sun Nov 24 14:53:17.946 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:17.947 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:17.949 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292592d0cf23a90f681c5e0') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259180cf23a90f681c05a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:17.951 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:17.951 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:17.953 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292592d0cf23a90f681c5e0') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259180cf23a90f681c05a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:17.993 [conn3643] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:53:17.994 [conn3643] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:53:17.996 [conn3643] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:53:17-5292592d18c2315fef764aee", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:34274", time: new Date(1385322797996), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, step1 of 6: 1, note: "aborted" } } Sun Nov 24 14:53:18.002 [conn10] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:53:18.003 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:53:18.105 [conn3643] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:453 111ms Sun Nov 24 14:53:18.105 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292592d18c2315fef764aec Sun Nov 24 14:53:18.107 [conn7] splitChunk accepted at version 3|9||529258b4c868398bb45e3755 Sun Nov 24 14:53:18.167 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:319854 160ms Sun Nov 24 14:53:18.624 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:681683 341ms Sun Nov 24 14:53:18.857 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:53:18-5292592e18c2315fef764aef", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322798857), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|9, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, lastmod: Timestamp 3000|10, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|11, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:53:19.471 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:610114 305ms Sun Nov 24 14:53:19.558 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:19.560 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259180cf23a90f681c05a') } -->> { : MaxKey } Sun Nov 24 14:53:19.562 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292592f0cf23a90f681c6c6') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259180cf23a90f681c05a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:19.664 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:53:19.664 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292592d0cf23a90f681c5e0') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259180cf23a90f681c05a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:146 reslen:37 1714ms Sun Nov 24 14:53:19.805 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:440373 220ms Sun Nov 24 14:53:19.865 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:19.867 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:20.258 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.5, filling with zeroes... Sun Nov 24 14:53:20.978 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:270 623ms Sun Nov 24 14:53:20.978 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:286 623ms Sun Nov 24 14:53:21.709 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1216380 608ms Sun Nov 24 14:53:21.746 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:21.748 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:22.164 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:672902 336ms Sun Nov 24 14:53:22.802 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1029817 515ms Sun Nov 24 14:53:23.607 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1368802 684ms Sun Nov 24 14:53:23.665 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:23.665 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:24.216 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:980966 491ms Sun Nov 24 14:53:25.509 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1079269 540ms Sun Nov 24 14:53:25.568 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:25.573 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:26.123 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:199 496ms Sun Nov 24 14:53:26.124 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:993135 497ms Sun Nov 24 14:53:26.804 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1113343 557ms Sun Nov 24 14:53:46.641 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.5, size: 2047MB, took 26.381 secs Sun Nov 24 14:53:47.561 [journal] old journal file will be removed: /shared/mongodb/data/db/dicom/dicom1/journal/j._0 Sun Nov 24 14:53:47.567 [journal] warning exception in dur::removeOldJournalFile /shared/mongodb/data/db/dicom/dicom1/journal/j._0: boost::filesystem::rename: No such file or directory: "/shared/mongodb/data/db/dicom/dicom1/journal/j._0", "/shared/mongodb/data/db/dicom/dicom1/journal/prealloc.0.temp" Sun Nov 24 14:53:47.567 [journal] old journal file will be removed: /shared/mongodb/data/db/dicom/dicom1/journal/j._1 Sun Nov 24 14:53:47.567 [journal] warning exception in dur::removeOldJournalFile /shared/mongodb/data/db/dicom/dicom1/journal/j._1: boost::filesystem::rename: No such file or directory: "/shared/mongodb/data/db/dicom/dicom1/journal/j._1", "/shared/mongodb/data/db/dicom/dicom1/journal/prealloc.0.temp" Sun Nov 24 14:53:47.801 [journal] DR101 latency warning on journal file open 233ms Sun Nov 24 14:53:47.895 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:47.896 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:47.898 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292594b0cf23a90f681ca95') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:47.902 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:47.903 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:47.905 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292594b0cf23a90f681ca95') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:48.008 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:53:48.009 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:53:48.227 [conn10] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:53:48.227 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:53:48.227 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292594b0cf23a90f681ca95') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:383 321ms Sun Nov 24 14:53:48.227 [conn3641] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:53:48.227 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:53:48.227 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292594b0cf23a90f681ca95') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:383 328ms Sun Nov 24 14:53:48.375 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:261909 131ms Sun Nov 24 14:53:48.576 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292594c18c2315fef764af2 Sun Nov 24 14:53:48.576 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:53:48-5292594c18c2315fef764af3", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322828576), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, from: "dicom1", to: "dicom12" } } Sun Nov 24 14:53:48.682 [conn15] moveChunk request accepted at version 3|11||529258b4c868398bb45e3755 Sun Nov 24 14:53:48.796 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:607057 303ms Sun Nov 24 14:53:48.798 [conn15] moveChunk number of documents: 166 Sun Nov 24 14:53:48.801 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:48.804 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:48.809 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:48.818 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:48.837 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:48.871 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:48.937 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:49.066 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:49.324 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:49.546 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1249877 625ms Sun Nov 24 14:53:49.648 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:49.649 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:49.649 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:49.650 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:49.651 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292594d0cf23a90f681cb77') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:49.653 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292594d0cf23a90f681cb77') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:49.838 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:50.247 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1165731 582ms Sun Nov 24 14:53:50.953 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:51.185 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:252 809ms Sun Nov 24 14:53:51.185 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:298 809ms Sun Nov 24 14:53:51.979 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:52.132 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1656831 829ms Sun Nov 24 14:53:52.763 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1036383 518ms Sun Nov 24 14:53:52.827 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:52.827 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:52.828 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:52.828 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:52.831 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259500cf23a90f681cc5d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:52.831 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259500cf23a90f681cc5d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:53.005 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 102, clonedBytes: 16384954, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:53.635 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:53.636 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:53.639 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259510cf23a90f681cc85') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:53.641 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:53.643 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:53.644 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259510cf23a90f681cc85') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:54.031 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 102, clonedBytes: 16384954, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:54.384 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1313120 656ms Sun Nov 24 14:53:55.057 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 103, clonedBytes: 16589686, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:55.445 [conn3634] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:43102 reslen:15372943 563ms Sun Nov 24 14:53:56.083 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 166, clonedBytes: 31962336, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:56.084 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1198112 599ms Sun Nov 24 14:53:56.163 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:56.164 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:56.165 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:56.165 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:53:56.168 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259540cf23a90f681cd72') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:56.168 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259540cf23a90f681cd72') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:53:57.109 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 166, clonedBytes: 31962336, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:57.115 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:248 914ms Sun Nov 24 14:53:57.117 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1829915 915ms Sun Nov 24 14:53:57.929 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:203 695ms Sun Nov 24 14:53:57.929 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:397 695ms Sun Nov 24 14:53:58.135 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 166, clonedBytes: 31962336, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:53:58.687 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1275253 637ms Sun Nov 24 14:53:58.689 [conn3634] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:39 reslen:51 652ms Sun Nov 24 14:53:58.689 [conn15] moveChunk setting version to: 4|0||529258b4c868398bb45e3755 Sun Nov 24 14:53:58.696 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:218983 88051ms Sun Nov 24 14:53:58.699 [conn3642] waiting till out of critical section Sun Nov 24 14:53:58.702 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 166, clonedBytes: 31962336, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:53:58.703 [conn15] moveChunk updating self version to: 4|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529258dc0cf23a90f681aef9') } -> { files_id: ObjectId('529258e80cf23a90f681b303') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:54:01.350 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:01-5292595918c2315fef764af4", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322841350), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, from: "dicom1", to: "dicom12" } } Sun Nov 24 14:54:01.350 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 3000|11, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:104 reslen:325 2650ms Sun Nov 24 14:54:01.494 [conn3642] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:257998 129ms Sun Nov 24 14:54:01.572 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:01.574 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:01.574 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:01.576 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259590cf23a90f681ce52') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:01.698 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259590cf23a90f681ce52') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:383 122ms Sun Nov 24 14:54:01.698 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:01.698 [conn3641] command admin.$cmd command: { splitVector: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 134217728, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) r:246079 reslen:129 124ms Sun Nov 24 14:54:02.027 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259590cf23a90f681ce52') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:02.519 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:254 378ms Sun Nov 24 14:54:02.519 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:323 378ms Sun Nov 24 14:54:02.988 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:690633 345ms Sun Nov 24 14:54:03.443 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:666646 333ms Sun Nov 24 14:54:03.499 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:03.500 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:03.502 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292595b0cf23a90f681cf5a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:03.503 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:03.504 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:03.507 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292595b0cf23a90f681cf5a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:03.890 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:658023 329ms Sun Nov 24 14:54:04.007 [conn10] request split points lookup for chunk dicomdb.fs.files { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:54:04.008 [conn3641] request split points lookup for chunk dicomdb.fs.files { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:54:04.009 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.files { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:54:04.010 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('5292595b0cf23a90f681cfc2') } ], shardId: "dicomdb.fs.files-_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:04.011 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.files { : ObjectId('529258d70cf23a90f681ad59') } -->> { : MaxKey } Sun Nov 24 14:54:04.011 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('5292595b0cf23a90f681cfc2') } ], shardId: "dicomdb.fs.files-_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:04.898 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:54:04.914 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:54:04.914 [conn15] forking for cleanup of chunk data Sun Nov 24 14:54:04.914 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:54:04.914 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:54:04.915 [cleanupOldData-5292595c18c2315fef764af7] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529258d80cf23a90f681ad65') } -> { files_id: ObjectId('529258dc0cf23a90f681aef9') }, # cursors remaining: 0 Sun Nov 24 14:54:04.936 [cleanupOldData-5292595c18c2315fef764af7] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529258d80cf23a90f681ad65') } -> { files_id: ObjectId('529258dc0cf23a90f681aef9') } Sun Nov 24 14:54:04.936 [cleanupOldData-5292595c18c2315fef764af7] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529258d80cf23a90f681ad65') } -> { files_id: ObjectId('529258dc0cf23a90f681aef9') } Sun Nov 24 14:54:04.964 [cleanupOldData-5292595c18c2315fef764af7] moveChunk deleted 166 documents for dicomdb.fs.chunks from { files_id: ObjectId('529258d80cf23a90f681ad65') } -> { files_id: ObjectId('529258dc0cf23a90f681aef9') } Sun Nov 24 14:54:05.073 [conn3641] could not acquire lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:54:05.073 [conn3641] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:54:05.073 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('5292595b0cf23a90f681cfc2') } ], shardId: "dicomdb.fs.files-_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:375 1062ms Sun Nov 24 14:54:05.229 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:439 155ms Sun Nov 24 14:54:06.104 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1519430 760ms Sun Nov 24 14:54:06.596 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:174 378ms Sun Nov 24 14:54:07.289 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:230 579ms Sun Nov 24 14:54:07.363 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:07.364 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:07.366 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292595f0cf23a90f681d09f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:07.386 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:07.387 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:07.389 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292595f0cf23a90f681d0a2') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:07.836 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:875561 438ms Sun Nov 24 14:54:08.403 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:223 455ms Sun Nov 24 14:54:08.867 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:705116 353ms Sun Nov 24 14:54:09.360 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:217 380ms Sun Nov 24 14:54:09.679 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:54:09.679 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:09-5292596118c2315fef764af8", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322849679), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, step1 of 6: 1, step2 of 6: 673, step3 of 6: 115, step4 of 6: 9460, step5 of 6: 6655, step6 of 6: 0 } } Sun Nov 24 14:54:09.679 [conn10] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292595c18c2315fef764af5 Sun Nov 24 14:54:09.682 [conn10] splitChunk accepted at version 2|1||529258bac868398bb45e3757 Sun Nov 24 14:54:09.860 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:772825 386ms Sun Nov 24 14:54:10.436 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:10.628 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:10.628 [conn3641] command admin.$cmd command: { splitVector: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 134217728, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 locks(micros) r:191710 reslen:129 191ms Sun Nov 24 14:54:10.630 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259620cf23a90f681d1ec') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:13.188 [DataFileSync] flushing mmaps took 23235ms for 10 files Sun Nov 24 14:54:13.191 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:13-5292596518c2315fef764afa", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322853191), what: "split", ns: "dicomdb.fs.files", details: { before: { min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: MaxKey }, lastmod: Timestamp 2000|1, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, lastmod: Timestamp 2000|2, lastmodEpoch: ObjectId('529258bac868398bb45e3757') }, right: { min: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, max: { _id: MaxKey }, lastmod: Timestamp 2000|3, lastmodEpoch: ObjectId('529258bac868398bb45e3757') } } } Sun Nov 24 14:54:13.191 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom12/node40.clus.cci.emory.edu:27012", fromShard: "dicom1", toShard: "dicom12", min: { files_id: ObjectId('529258d80cf23a90f681ad65') }, max: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258d80cf23a90f681ad65')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:123 r:1109 w:48 reslen:37 25183ms Sun Nov 24 14:54:13.578 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292596218c2315fef764af9 Sun Nov 24 14:54:13.583 [conn3641] splitChunk accepted at version 4|1||529258b4c868398bb45e3755 Sun Nov 24 14:54:13.685 [conn10] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:54:13.685 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.files", keyPattern: { _id: 1.0 }, min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: MaxKey }, from: "dicom1", splitKeys: [ { _id: ObjectId('5292595b0cf23a90f681cfc2') } ], shardId: "dicomdb.fs.files-_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:163 reslen:37 9674ms Sun Nov 24 14:54:13.685 [conn3641] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:13-5292596518c2315fef764afb", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:34240", time: new Date(1385322853685), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, lastmod: Timestamp 3000|11, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, lastmod: Timestamp 4000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, lastmod: Timestamp 4000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:54:13.706 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:13.707 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292592d0cf23a90f681c5e0') } -->> { : MaxKey } Sun Nov 24 14:54:13.709 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259650cf23a90f681d1ef') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:13.933 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:54:13.933 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259620cf23a90f681d1ec') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:123 reslen:37 3303ms Sun Nov 24 14:54:14.134 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:336273 168ms Sun Nov 24 14:54:14.199 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:14.204 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:14.722 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:210 182ms Sun Nov 24 14:54:14.722 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:363 182ms Sun Nov 24 14:54:15.007 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:227 171ms Sun Nov 24 14:54:15.008 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:342317 171ms Sun Nov 24 14:54:15.295 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:273 171ms Sun Nov 24 14:54:15.296 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:343392 172ms Sun Nov 24 14:54:15.296 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:54 reslen:3141 112ms Sun Nov 24 14:54:15.330 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:54:15.331 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom13/node40.clus.cci.emory.edu:27013", fromShard: "dicom1", toShard: "dicom13", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258dc0cf23a90f681aef9')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:54:15.388 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:15.395 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:15.725 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:630912 315ms Sun Nov 24 14:54:16.125 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:229 284ms Sun Nov 24 14:54:16.125 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:569265 285ms Sun Nov 24 14:54:16.381 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292596718c2315fef764afc Sun Nov 24 14:54:16.382 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:16-5292596818c2315fef764afd", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322856382), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, from: "dicom1", to: "dicom13" } } Sun Nov 24 14:54:16.449 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:417532 209ms Sun Nov 24 14:54:16.488 [conn15] moveChunk request accepted at version 4|3||529258b4c868398bb45e3755 Sun Nov 24 14:54:16.493 [conn15] moveChunk number of documents: 621 Sun Nov 24 14:54:16.493 [conn15] starting new replica set monitor for replica set dicom13 with seed of node40.clus.cci.emory.edu:27013 Sun Nov 24 14:54:16.494 [conn15] successfully connected to seed node40.clus.cci.emory.edu:27013 for replica set dicom13 Sun Nov 24 14:54:16.495 [conn15] changing hosts to { 0: "node40.clus.cci.emory.edu:27013" } from dicom13/ Sun Nov 24 14:54:16.495 [conn15] trying to add new host node40.clus.cci.emory.edu:27013 to replica set dicom13 Sun Nov 24 14:54:16.495 [conn15] successfully connected to new host node40.clus.cci.emory.edu:27013 in replica set dicom13 Sun Nov 24 14:54:16.497 [conn15] Primary for replica set dicom13 changed to node40.clus.cci.emory.edu:27013 Sun Nov 24 14:54:16.497 [conn15] replica set monitor for replica set dicom13 started, address is dicom13/node40.clus.cci.emory.edu:27013 Sun Nov 24 14:54:16.502 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.505 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.511 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.521 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.538 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.572 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.636 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.765 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:16.901 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:673277 337ms Sun Nov 24 14:54:17.022 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:17.190 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:351007 175ms Sun Nov 24 14:54:17.190 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:17.191 [conn3641] command admin.$cmd command: { splitVector: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 134217728, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 locks(micros) r:660 reslen:69 179ms Sun Nov 24 14:54:17.199 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:17.536 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:18.165 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1019531 510ms Sun Nov 24 14:54:18.561 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:19.597 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:20.622 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:21.430 [conn3647] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:66841 reslen:16771393 3191ms Sun Nov 24 14:54:21.647 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 160, clonedBytes: 16770652, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:21.731 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:502561 251ms Sun Nov 24 14:54:21.799 [conn3647] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34852 reslen:16572783 226ms Sun Nov 24 14:54:21.830 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:21.837 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:22.156 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:200 311ms Sun Nov 24 14:54:22.156 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:292 310ms Sun Nov 24 14:54:22.672 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 160, clonedBytes: 16770652, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:22.810 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1067113 534ms Sun Nov 24 14:54:23.346 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:844763 422ms Sun Nov 24 14:54:23.731 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 252, clonedBytes: 33343026, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:24.007 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1082877 541ms Sun Nov 24 14:54:24.475 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:224 348ms Sun Nov 24 14:54:24.475 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:263 348ms Sun Nov 24 14:54:24.520 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:24.527 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:24.757 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 385, clonedBytes: 49990000, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:24.864 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:550591 275ms Sun Nov 24 14:54:25.168 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:371872 186ms Sun Nov 24 14:54:25.225 [conn3647] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33858 reslen:16519271 145ms Sun Nov 24 14:54:25.571 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:580552 290ms Sun Nov 24 14:54:25.782 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 448, clonedBytes: 66508978, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:26.247 [conn3647] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33926 reslen:16537727 142ms Sun Nov 24 14:54:26.640 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:26.640 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:26.806 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 511, clonedBytes: 83027956, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:27.119 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:206 411ms Sun Nov 24 14:54:27.120 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:135 412ms Sun Nov 24 14:54:27.695 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:915519 458ms Sun Nov 24 14:54:27.751 [conn3647] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:42363 reslen:9701811 476ms Sun Nov 24 14:54:27.832 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 584, clonedBytes: 99565350, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:28.063 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:214 254ms Sun Nov 24 14:54:28.063 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:278 254ms Sun Nov 24 14:54:28.558 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:758527 379ms Sun Nov 24 14:54:28.561 [conn3647] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:44 reslen:51 209ms Sun Nov 24 14:54:28.858 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 621, clonedBytes: 109266972, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:29.170 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:994808 497ms Sun Nov 24 14:54:29.173 [conn3647] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:18 reslen:51 228ms Sun Nov 24 14:54:29.274 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:29.275 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:29.276 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259750cf23a90f681daff') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:29.285 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:29.286 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:29.286 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259750cf23a90f681daff') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:29.631 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:276 345ms Sun Nov 24 14:54:29.631 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:223 342ms Sun Nov 24 14:54:29.884 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 621, clonedBytes: 109266972, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:30.222 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:953767 477ms Sun Nov 24 14:54:30.222 [conn3647] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:48 reslen:51 484ms Sun Nov 24 14:54:30.222 [conn15] moveChunk setting version to: 5|0||529258b4c868398bb45e3755 Sun Nov 24 14:54:30.226 [conn3642] waiting till out of critical section Sun Nov 24 14:54:30.234 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 621, clonedBytes: 109266972, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:54:30.234 [conn15] moveChunk updating self version to: 5|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529258e80cf23a90f681b303') } -> { files_id: ObjectId('529258ee0cf23a90f681b56f') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:54:31.702 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:31-5292597718c2315fef764afe", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322871702), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, from: "dicom1", to: "dicom13" } } Sun Nov 24 14:54:31.702 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 4000|3, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:69 reslen:325 1476ms Sun Nov 24 14:54:31.968 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:430330 215ms Sun Nov 24 14:54:32.012 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:32.013 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:32.015 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:32.015 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259780cf23a90f681db8e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:32.016 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:32.018 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259780cf23a90f681db8e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:32.901 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1634193 817ms Sun Nov 24 14:54:34.095 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:2151150 1076ms Sun Nov 24 14:54:34.839 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1256044 628ms Sun Nov 24 14:54:35.144 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:54:35.765 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:207 806ms Sun Nov 24 14:54:35.765 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:259 806ms Sun Nov 24 14:54:35.765 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:54:35.765 [conn15] forking for cleanup of chunk data Sun Nov 24 14:54:35.765 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:54:35.765 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:54:35.765 [cleanupOldData-5292597b18c2315fef764aff] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529258dc0cf23a90f681aef9') } -> { files_id: ObjectId('529258e80cf23a90f681b303') }, # cursors remaining: 0 Sun Nov 24 14:54:35.787 [cleanupOldData-5292597b18c2315fef764aff] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529258dc0cf23a90f681aef9') } -> { files_id: ObjectId('529258e80cf23a90f681b303') } Sun Nov 24 14:54:35.828 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:35.828 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:35.988 [cleanupOldData-5292597b18c2315fef764aff] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529258dc0cf23a90f681aef9') } -> { files_id: ObjectId('529258e80cf23a90f681b303') } Sun Nov 24 14:54:35.990 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:35.990 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:35.990 [conn7] command admin.$cmd command: { splitVector: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 134217728, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) r:322011 reslen:129 162ms Sun Nov 24 14:54:35.991 [conn3643] command admin.$cmd command: { splitVector: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 134217728, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) r:321368 reslen:129 162ms Sun Nov 24 14:54:35.992 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292597b0cf23a90f681dcec') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:35.994 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292597b0cf23a90f681dcec') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:36.190 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:258 194ms Sun Nov 24 14:54:36.190 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:228 193ms Sun Nov 24 14:54:36.535 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:455367 228ms Sun Nov 24 14:54:36.694 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:54:36.694 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:36-5292597c18c2315fef764b00", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322876694), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, step1 of 6: 1, step2 of 6: 1156, step3 of 6: 11, step4 of 6: 13384, step5 of 6: 5881, step6 of 6: 0 } } Sun Nov 24 14:54:36.866 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:264 211ms Sun Nov 24 14:54:36.866 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:257 211ms Sun Nov 24 14:54:36.946 [cleanupOldData-5292597b18c2315fef764aff] moveChunk deleted 621 documents for dicomdb.fs.chunks from { files_id: ObjectId('529258dc0cf23a90f681aef9') } -> { files_id: ObjectId('529258e80cf23a90f681b303') } Sun Nov 24 14:54:37.117 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom13/node40.clus.cci.emory.edu:27013", fromShard: "dicom1", toShard: "dicom13", min: { files_id: ObjectId('529258dc0cf23a90f681aef9') }, max: { files_id: ObjectId('529258e80cf23a90f681b303') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258dc0cf23a90f681aef9')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 numYields: 2 locks(micros) W:201 r:2560 w:62 reslen:37 21786ms Sun Nov 24 14:54:37.123 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:54:37.124 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:54:37.235 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:493760 247ms Sun Nov 24 14:54:37.572 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:37.573 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:37.575 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292597d0cf23a90f681de3f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:37.577 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:37.578 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:37.579 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292597d0cf23a90f681de3f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:37.584 [conn15] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292597d18c2315fef764b01 Sun Nov 24 14:54:37.585 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:37-5292597d18c2315fef764b04", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322877585), what: "moveChunk.start", ns: "dicomdb.fs.files", details: { min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:54:37.644 [conn3643] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('5292596718c2315fef764afc') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn3643:179229628", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322877575), why: "split-{ files_id: ObjectId('529259620cf23a90f681d1ec') }", ts: ObjectId('5292597d18c2315fef764b02') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322730000|9, connectionId: 28, waited: 24, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322877000|7, connectionId: 61, waited: 0, err: null, ok: 1.0 } Sun Nov 24 14:54:37.644 [conn7] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('5292596718c2315fef764afc') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322877580), why: "split-{ files_id: ObjectId('529259620cf23a90f681d1ec') }", ts: ObjectId('5292597d18c2315fef764b03') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322877000|7, connectionId: 29, waited: 25, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322853000|3, connectionId: 62, waited: 0, err: null, ok: 1.0 } Sun Nov 24 14:54:37.644 [conn15] moveChunk request accepted at version 2|3||529258bac868398bb45e3757 Sun Nov 24 14:54:37.645 [conn3643] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:54:37.645 [conn3643] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:54:37.647 [conn7] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:54:37.651 [conn15] moveChunk number of documents: 2022 Sun Nov 24 14:54:37.654 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.659 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.664 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:37.664 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.665 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259620cf23a90f681d1ec') } -->> { : MaxKey } Sun Nov 24 14:54:37.666 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292597d0cf23a90f681de42') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:37.674 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "clone", counts: { cloned: 66, clonedBytes: 19109, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.692 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "clone", counts: { cloned: 245, clonedBytes: 71041, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.726 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "clone", counts: { cloned: 964, clonedBytes: 280184, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.792 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "clone", counts: { cloned: 1383, clonedBytes: 401694, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.821 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292597d18c2315fef764b03 Sun Nov 24 14:54:37.823 [conn7] splitChunk accepted at version 5|1||529258b4c868398bb45e3755 Sun Nov 24 14:54:37.923 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "clone", counts: { cloned: 2022, clonedBytes: 587201, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:37.927 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:37-5292597d18c2315fef764b05", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322877927), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, lastmod: Timestamp 4000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, lastmod: Timestamp 5000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, lastmod: Timestamp 5000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:54:37.980 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:254761 127ms Sun Nov 24 14:54:37.980 [conn3638] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:48 reslen:51 122ms Sun Nov 24 14:54:38.180 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "steady", counts: { cloned: 2022, clonedBytes: 587201, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:38.297 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:410653 205ms Sun Nov 24 14:54:38.297 [conn15] moveChunk setting version to: 3|0||529258bac868398bb45e3757 Sun Nov 24 14:54:38.297 [conn3638] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:63 reslen:51 206ms Sun Nov 24 14:54:38.299 [conn3642] waiting till out of critical section Sun Nov 24 14:54:38.315 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, shardKeyPattern: { _id: 1.0 }, state: "done", counts: { cloned: 2022, clonedBytes: 587201, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:54:38.316 [conn15] moveChunk updating self version to: 3|1||529258bac868398bb45e3757 through { _id: ObjectId('5292595b0cf23a90f681cfc2') } -> { _id: MaxKey } for collection 'dicomdb.fs.files' Sun Nov 24 14:54:38.770 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:54:38.770 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:38-5292597e18c2315fef764b06", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322878770), what: "moveChunk.commit", ns: "dicomdb.fs.files", details: { min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, from: "dicom1", to: "dicom11" } } Sun Nov 24 14:54:38.770 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292597d0cf23a90f681de3f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:130 reslen:37 1190ms Sun Nov 24 14:54:38.770 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.files", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 2000|3, versionEpoch: ObjectId('529258bac868398bb45e3757'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:91 reslen:323 471ms Sun Nov 24 14:54:38.862 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:38.870 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:38.876 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:54:38.876 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:54:38.876 [conn15] forking for cleanup of chunk data Sun Nov 24 14:54:38.876 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:54:38.876 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:54:38.877 [cleanupOldData-5292597e18c2315fef764b07] (start) waiting to cleanup dicomdb.fs.files from { _id: ObjectId('529258d70cf23a90f681ad59') } -> { _id: ObjectId('5292595b0cf23a90f681cfc2') }, # cursors remaining: 0 Sun Nov 24 14:54:38.898 [cleanupOldData-5292597e18c2315fef764b07] waiting to remove documents for dicomdb.fs.files from { _id: ObjectId('529258d70cf23a90f681ad59') } -> { _id: ObjectId('5292595b0cf23a90f681cfc2') } Sun Nov 24 14:54:38.898 [cleanupOldData-5292597e18c2315fef764b07] moveChunk starting delete for: dicomdb.fs.files from { _id: ObjectId('529258d70cf23a90f681ad59') } -> { _id: ObjectId('5292595b0cf23a90f681cfc2') } Sun Nov 24 14:54:38.947 [conn15] distributed lock 'dicomdb.fs.files/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:54:38.947 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:38-5292597e18c2315fef764b08", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322878947), what: "moveChunk.from", ns: "dicomdb.fs.files", details: { min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, step1 of 6: 1, step2 of 6: 520, step3 of 6: 7, step4 of 6: 529, step5 of 6: 695, step6 of 6: 0 } } Sun Nov 24 14:54:39.055 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:337314 169ms Sun Nov 24 14:54:39.088 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.files", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom11/node40.clus.cci.emory.edu:27011", fromShard: "dicom1", toShard: "dicom11", min: { _id: ObjectId('529258d70cf23a90f681ad59') }, max: { _id: ObjectId('5292595b0cf23a90f681cfc2') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.files-_id_ObjectId('529258d70cf23a90f681ad59')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:88 r:9463 w:36 reslen:37 1964ms Sun Nov 24 14:54:39.346 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:351766 176ms Sun Nov 24 14:54:39.679 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:432619 216ms Sun Nov 24 14:54:40.332 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:448440 224ms Sun Nov 24 14:54:40.373 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:40.373 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:40.670 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:435238 218ms Sun Nov 24 14:54:41.034 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:268 244ms Sun Nov 24 14:54:41.035 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:489656 245ms Sun Nov 24 14:54:41.109 [cleanupOldData-5292597e18c2315fef764b07] moveChunk deleted 2022 documents for dicomdb.fs.files from { _id: ObjectId('529258d70cf23a90f681ad59') } -> { _id: ObjectId('5292595b0cf23a90f681cfc2') } Sun Nov 24 14:54:41.362 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:412461 206ms Sun Nov 24 14:54:41.366 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:157 reslen:3141 192ms Sun Nov 24 14:54:41.399 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:54:41.401 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom14/node40.clus.cci.emory.edu:27014", fromShard: "dicom1", toShard: "dicom14", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:54:41.703 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:442419 221ms Sun Nov 24 14:54:41.765 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:41.767 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:41.944 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292598118c2315fef764b09 Sun Nov 24 14:54:41.944 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:41-5292598118c2315fef764b0a", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322881944), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, from: "dicom1", to: "dicom14" } } Sun Nov 24 14:54:42.012 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:385623 193ms Sun Nov 24 14:54:42.051 [conn15] moveChunk request accepted at version 5|3||529258b4c868398bb45e3755 Sun Nov 24 14:54:42.056 [conn15] moveChunk number of documents: 363 Sun Nov 24 14:54:42.056 [conn15] starting new replica set monitor for replica set dicom14 with seed of node40.clus.cci.emory.edu:27014 Sun Nov 24 14:54:42.057 [conn15] successfully connected to seed node40.clus.cci.emory.edu:27014 for replica set dicom14 Sun Nov 24 14:54:42.057 [conn15] changing hosts to { 0: "node40.clus.cci.emory.edu:27014" } from dicom14/ Sun Nov 24 14:54:42.057 [conn15] trying to add new host node40.clus.cci.emory.edu:27014 to replica set dicom14 Sun Nov 24 14:54:42.058 [conn15] successfully connected to new host node40.clus.cci.emory.edu:27014 in replica set dicom14 Sun Nov 24 14:54:42.060 [conn15] Primary for replica set dicom14 changed to node40.clus.cci.emory.edu:27014 Sun Nov 24 14:54:42.060 [conn15] replica set monitor for replica set dicom14 started, address is dicom14/node40.clus.cci.emory.edu:27014 Sun Nov 24 14:54:42.066 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.070 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.076 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.086 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.104 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.138 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.204 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.334 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:42.494 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:731769 366ms Sun Nov 24 14:54:42.592 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:43.106 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:43.450 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:695725 348ms Sun Nov 24 14:54:43.532 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:43.535 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:43.988 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:284 421ms Sun Nov 24 14:54:43.989 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:842294 421ms Sun Nov 24 14:54:44.133 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:44.343 [conn3651] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:31509 reslen:16626061 169ms Sun Nov 24 14:54:44.830 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:840892 420ms Sun Nov 24 14:54:44.884 [conn3651] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:32027 reslen:16702504 365ms Sun Nov 24 14:54:45.159 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 110, clonedBytes: 16625570, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:45.399 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:379 451ms Sun Nov 24 14:54:45.399 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:199 450ms Sun Nov 24 14:54:45.484 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:45.488 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:46.069 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1091033 545ms Sun Nov 24 14:54:46.111 [conn3651] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:31073 reslen:16519271 425ms Sun Nov 24 14:54:46.185 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 239, clonedBytes: 33327488, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:46.696 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1025823 513ms Sun Nov 24 14:54:47.193 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:272 375ms Sun Nov 24 14:54:47.193 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:362 375ms Sun Nov 24 14:54:47.210 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 302, clonedBytes: 49846466, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:47.611 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:207 294ms Sun Nov 24 14:54:47.612 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:589914 295ms Sun Nov 24 14:54:47.613 [conn3651] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:64 reslen:51 125ms Sun Nov 24 14:54:47.648 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:47.649 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:47.650 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:47.650 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:47.652 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259870cf23a90f681e542') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:47.654 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259870cf23a90f681e542') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:48.181 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:910702 455ms Sun Nov 24 14:54:48.235 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 363, clonedBytes: 65841032, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:54:48.236 [conn15] moveChunk setting version to: 6|0||529258b4c868398bb45e3755 Sun Nov 24 14:54:48.248 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 363, clonedBytes: 65841032, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:54:48.249 [conn15] moveChunk updating self version to: 6|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529258ee0cf23a90f681b56f') } -> { files_id: ObjectId('529258f60cf23a90f681b682') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:54:48.509 [conn3642] waiting till out of critical section Sun Nov 24 14:54:56.105 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:54:56-5292599018c2315fef764b0b", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322896105), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, from: "dicom1", to: "dicom14" } } Sun Nov 24 14:54:56.105 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 5000|3, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:118 reslen:325 7595ms Sun Nov 24 14:54:56.408 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:446715 223ms Sun Nov 24 14:54:56.445 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:56.446 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:56.448 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:56.449 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:56.449 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259900cf23a90f681e5c4') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:56.452 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259900cf23a90f681e5c4') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:56.815 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:593855 297ms Sun Nov 24 14:54:57.397 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:191 461ms Sun Nov 24 14:54:57.398 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:923253 462ms Sun Nov 24 14:54:58.603 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:766671 383ms Sun Nov 24 14:54:58.640 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:58.642 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:58.644 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:58.644 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259920cf23a90f681e6aa') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:58.645 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:54:58.648 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259920cf23a90f681e6aa') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:54:59.120 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:797269 399ms Sun Nov 24 14:54:59.664 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:843935 422ms Sun Nov 24 14:55:00.309 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:243 522ms Sun Nov 24 14:55:00.310 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1045665 523ms Sun Nov 24 14:55:00.378 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:55:07.943 [LockPinger] trying to delete 1 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:55:09.595 [DataFileSync] flushing mmaps took 19641ms for 10 files Sun Nov 24 14:55:09.595 [FileAllocator] allocating new datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.6, filling with zeroes... Sun Nov 24 14:55:09.598 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:9280464 9279ms Sun Nov 24 14:55:09.598 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:55:09.598 [conn15] forking for cleanup of chunk data Sun Nov 24 14:55:09.598 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:55:09.598 [cleanupOldData-5292599d18c2315fef764b0c] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529258e80cf23a90f681b303') } -> { files_id: ObjectId('529258ee0cf23a90f681b56f') }, # cursors remaining: 0 Sun Nov 24 14:55:09.600 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:55:09.602 [LockPinger] handled late remove of old distributed lock with ts 5292597d18c2315fef764b02 Sun Nov 24 14:55:09.619 [cleanupOldData-5292599d18c2315fef764b0c] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529258e80cf23a90f681b303') } -> { files_id: ObjectId('529258ee0cf23a90f681b56f') } Sun Nov 24 14:55:09.619 [cleanupOldData-5292599d18c2315fef764b0c] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529258e80cf23a90f681b303') } -> { files_id: ObjectId('529258ee0cf23a90f681b56f') } Sun Nov 24 14:55:09.717 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:196 111ms Sun Nov 24 14:55:09.717 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:230 111ms Sun Nov 24 14:55:09.736 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:55:09.736 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:55:09-5292599d18c2315fef764b0d", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322909736), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, step1 of 6: 1, step2 of 6: 650, step3 of 6: 11, step4 of 6: 6173, step5 of 6: 21362, step6 of 6: 0 } } Sun Nov 24 14:55:09.756 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:09.756 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:09.757 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:09.757 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:09.760 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292599d0cf23a90f681e77a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:09.760 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292599d0cf23a90f681e77a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:09.799 [cleanupOldData-5292599d18c2315fef764b0c] moveChunk deleted 363 documents for dicomdb.fs.chunks from { files_id: ObjectId('529258e80cf23a90f681b303') } -> { files_id: ObjectId('529258ee0cf23a90f681b56f') } Sun Nov 24 14:55:38.481 [FileAllocator] done allocating datafile /shared/mongodb/data/db/dicom/dicom1/dicomdb/dicomdb.6, size: 2047MB, took 28.884 secs Sun Nov 24 14:55:39.649 [conn3643] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('5292598118c2315fef764b09') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn3643:179229628", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322909761), why: "split-{ files_id: ObjectId('5292597d0cf23a90f681de3f') }", ts: ObjectId('5292599d18c2315fef764b0f') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322730000|9, connectionId: 28, waited: 18, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322939000|2, connectionId: 61, waited: 32, err: null, ok: 1.0 } Sun Nov 24 14:55:39.649 [conn7] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('5292598118c2315fef764b09') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322909760), why: "split-{ files_id: ObjectId('5292597d0cf23a90f681de3f') }", ts: ObjectId('5292599d18c2315fef764b0e') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322939000|2, connectionId: 42, waited: 19, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322909000|1, connectionId: 75, waited: 32, err: null, ok: 1.0 } Sun Nov 24 14:55:39.649 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom14/node40.clus.cci.emory.edu:27014", fromShard: "dicom1", toShard: "dicom14", min: { files_id: ObjectId('529258e80cf23a90f681b303') }, max: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258e80cf23a90f681b303')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:253 r:1619 w:34 reslen:37 58249ms Sun Nov 24 14:55:39.650 [conn7] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:55:39.650 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:55:39.650 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292599d0cf23a90f681e77a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:382 29890ms Sun Nov 24 14:55:39.654 [conn3643] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:55:39.673 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:39.674 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:39.676 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259bb0cf23a90f681e783') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:39.913 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:372887 186ms Sun Nov 24 14:55:40.186 [conn3643] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 5292599d18c2315fef764b0f Sun Nov 24 14:55:40.205 [conn3643] splitChunk accepted at version 6|1||529258b4c868398bb45e3755 Sun Nov 24 14:55:40.354 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:649839 324ms Sun Nov 24 14:55:40.503 [LockPinger] cluster node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020 pinged successfully at Sun Nov 24 14:55:39 2013 by distributed lock pinger 'node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020/node40.clus.cci.emory.edu:27001:1385321687:2071045908', sleeping for 30000ms Sun Nov 24 14:55:40.503 [LockPinger] trying to delete 1 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:55:40.503 [conn3643] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:55:40-529259bc18c2315fef764b10", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:34274", time: new Date(1385322940503), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, lastmod: Timestamp 5000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, lastmod: Timestamp 6000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, lastmod: Timestamp 6000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:55:40.994 [LockPinger] handled late remove of old distributed lock with ts 5292599d18c2315fef764b0e Sun Nov 24 14:55:41.231 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:570121 285ms Sun Nov 24 14:55:41.304 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:41.305 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292597d0cf23a90f681de3f') } -->> { : MaxKey } Sun Nov 24 14:55:41.307 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259bd0cf23a90f681e869') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:41.602 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:514123 257ms Sun Nov 24 14:55:41.654 [conn3643] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:55:41.654 [conn3643] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('5292599d0cf23a90f681e77a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292597d0cf23a90f681de3f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:155 reslen:37 31893ms Sun Nov 24 14:55:41.711 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:42.123 [journal] old journal file will be removed: /shared/mongodb/data/db/dicom/dicom1/journal/j._2 Sun Nov 24 14:55:42.241 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1048715 524ms Sun Nov 24 14:55:42.242 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:55 reslen:3141 446ms Sun Nov 24 14:55:42.242 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:42.242 [conn3643] command admin.$cmd command: { splitVector: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 134217728, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 locks(micros) r:447 reslen:69 525ms Sun Nov 24 14:55:42.280 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:55:42.281 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom15/node40.clus.cci.emory.edu:27015", fromShard: "dicom1", toShard: "dicom15", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:55:42.532 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259be18c2315fef764b11 Sun Nov 24 14:55:42.532 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:55:42-529259be18c2315fef764b12", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322942532), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, from: "dicom1", to: "dicom15" } } Sun Nov 24 14:55:42.693 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:667123 334ms Sun Nov 24 14:55:42.779 [conn15] moveChunk request accepted at version 6|3||529258b4c868398bb45e3755 Sun Nov 24 14:55:42.784 [conn15] moveChunk number of documents: 165 Sun Nov 24 14:55:42.784 [conn15] starting new replica set monitor for replica set dicom15 with seed of node40.clus.cci.emory.edu:27015 Sun Nov 24 14:55:42.785 [conn15] successfully connected to seed node40.clus.cci.emory.edu:27015 for replica set dicom15 Sun Nov 24 14:55:42.787 [conn15] changing hosts to { 0: "node40.clus.cci.emory.edu:27015" } from dicom15/ Sun Nov 24 14:55:42.787 [conn15] trying to add new host node40.clus.cci.emory.edu:27015 to replica set dicom15 Sun Nov 24 14:55:42.788 [conn15] successfully connected to new host node40.clus.cci.emory.edu:27015 in replica set dicom15 Sun Nov 24 14:55:42.791 [conn15] Primary for replica set dicom15 changed to node40.clus.cci.emory.edu:27015 Sun Nov 24 14:55:42.792 [conn15] replica set monitor for replica set dicom15 started, address is dicom15/node40.clus.cci.emory.edu:27015 Sun Nov 24 14:55:42.796 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:42.799 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:42.804 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:42.813 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:42.830 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:42.863 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:42.928 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:43.057 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:43.273 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:923716 462ms Sun Nov 24 14:55:43.314 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:43.827 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:44.275 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1767740 884ms Sun Nov 24 14:55:44.286 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:44.286 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:44.721 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:660430 330ms Sun Nov 24 14:55:44.853 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:45.775 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:610722 305ms Sun Nov 24 14:55:45.813 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:45.813 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:45.846 [conn3655] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33433 reslen:16612507 253ms Sun Nov 24 14:55:45.879 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:46.356 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:202 466ms Sun Nov 24 14:55:46.357 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:932090 466ms Sun Nov 24 14:55:46.905 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 114, clonedBytes: 16611996, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:47.642 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1424774 712ms Sun Nov 24 14:55:47.645 [conn3655] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:61 reslen:51 353ms Sun Nov 24 14:55:47.730 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:47.736 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:47.931 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 165, clonedBytes: 29705122, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:55:48.072 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:622090 311ms Sun Nov 24 14:55:48.074 [conn15] moveChunk setting version to: 7|0||529258b4c868398bb45e3755 Sun Nov 24 14:55:48.075 [conn3655] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34 reslen:51 195ms Sun Nov 24 14:55:48.075 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:262523 51938ms Sun Nov 24 14:55:48.078 [conn3642] waiting till out of critical section Sun Nov 24 14:55:48.088 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 165, clonedBytes: 29705122, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:55:48.088 [conn15] moveChunk updating self version to: 7|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529258f60cf23a90f681b682') } -> { files_id: ObjectId('529258fe0cf23a90f681b98f') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:55:48.781 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:55:48-529259c418c2315fef764b13", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322948781), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, from: "dicom1", to: "dicom15" } } Sun Nov 24 14:55:48.781 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 6000|3, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:95 reslen:325 703ms Sun Nov 24 14:55:48.862 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:48.865 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:49.027 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:55:49.157 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:55:49.157 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:508061 254ms Sun Nov 24 14:55:49.157 [conn15] forking for cleanup of chunk data Sun Nov 24 14:55:49.157 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:55:49.157 [cleanupOldData-529259c518c2315fef764b14] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529258ee0cf23a90f681b56f') } -> { files_id: ObjectId('529258f60cf23a90f681b682') }, # cursors remaining: 0 Sun Nov 24 14:55:49.159 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:55:49.179 [cleanupOldData-529259c518c2315fef764b14] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529258ee0cf23a90f681b56f') } -> { files_id: ObjectId('529258f60cf23a90f681b682') } Sun Nov 24 14:55:49.179 [cleanupOldData-529259c518c2315fef764b14] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529258ee0cf23a90f681b56f') } -> { files_id: ObjectId('529258f60cf23a90f681b682') } Sun Nov 24 14:55:49.388 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:55:49.388 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:55:49-529259c518c2315fef764b15", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322949388), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, step1 of 6: 1, step2 of 6: 498, step3 of 6: 14, step4 of 6: 5137, step5 of 6: 1225, step6 of 6: 0 } } Sun Nov 24 14:55:49.696 [cleanupOldData-529259c518c2315fef764b14] moveChunk deleted 165 documents for dicomdb.fs.chunks from { files_id: ObjectId('529258ee0cf23a90f681b56f') } -> { files_id: ObjectId('529258f60cf23a90f681b682') } Sun Nov 24 14:55:49.739 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom15/node40.clus.cci.emory.edu:27015", fromShard: "dicom1", toShard: "dicom15", min: { files_id: ObjectId('529258ee0cf23a90f681b56f') }, max: { files_id: ObjectId('529258f60cf23a90f681b682') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258ee0cf23a90f681b56f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:144 r:1121 w:83 reslen:37 7458ms Sun Nov 24 14:55:50.109 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:621253 311ms Sun Nov 24 14:55:50.207 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:50.207 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:50.209 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:50.209 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:50.211 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259c60cf23a90f681ec8b') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292599d0cf23a90f681e77a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:50.212 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259c60cf23a90f681ec8b') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292599d0cf23a90f681e77a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:52.609 [conn3643] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529259be18c2315fef764b11') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn3643:179229628", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322950212), why: "split-{ files_id: ObjectId('5292599d0cf23a90f681e77a') }", ts: ObjectId('529259c618c2315fef764b16') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322881000|3, connectionId: 29, waited: 50, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322952000|2, connectionId: 62, waited: 112, err: null, ok: 1.0 } Sun Nov 24 14:55:52.609 [conn7] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529259be18c2315fef764b11') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn7:1837904310", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322950213), why: "split-{ files_id: ObjectId('5292599d0cf23a90f681e77a') }", ts: ObjectId('529259c618c2315fef764b17') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322952000|2, connectionId: 42, waited: 50, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322909000|1, connectionId: 75, waited: 112, err: null, ok: 1.0 } Sun Nov 24 14:55:52.610 [conn3643] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:55:52.610 [conn3643] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:55:52.610 [conn3643] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259c60cf23a90f681ec8b') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292599d0cf23a90f681e77a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:380 2399ms Sun Nov 24 14:55:52.611 [conn7] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:55:53.121 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:870380 435ms Sun Nov 24 14:55:53.774 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1074987 538ms Sun Nov 24 14:55:54.387 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:993952 497ms Sun Nov 24 14:55:54.779 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:554372 277ms Sun Nov 24 14:55:54.853 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:54.854 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:54.857 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259ca0cf23a90f681ed72') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292599d0cf23a90f681e77a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:55.141 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259c618c2315fef764b17 Sun Nov 24 14:55:55.144 [conn7] splitChunk accepted at version 7|1||529258b4c868398bb45e3755 Sun Nov 24 14:55:55.192 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:597665 299ms Sun Nov 24 14:55:55.229 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:55:55.230 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom16/node41.clus.cci.emory.edu:27001", fromShard: "dicom1", toShard: "dicom16", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258f60cf23a90f681b682')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:55:55.232 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:55:55-529259cb18c2315fef764b18", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322955232), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, step1 of 6: 1, note: "aborted" } } Sun Nov 24 14:55:56.563 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1247651 623ms Sun Nov 24 14:55:57.556 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:261 876ms Sun Nov 24 14:55:57.638 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:57.640 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:55:57.642 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259cd0cf23a90f681ee58') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292599d0cf23a90f681e77a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:55:58.194 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1048648 524ms Sun Nov 24 14:55:58.376 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom16/node41.clus.cci.emory.edu:27001", fromShard: "dicom1", toShard: "dicom16", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258f60cf23a90f681b682')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 reslen:453 3147ms Sun Nov 24 14:55:58.376 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:55:58-529259ce18c2315fef764b19", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322958376), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, lastmod: Timestamp 6000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, lastmod: Timestamp 7000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, lastmod: Timestamp 7000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:55:58.588 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:556956 279ms Sun Nov 24 14:55:59.345 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1279530 640ms Sun Nov 24 14:55:59.860 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:806532 403ms Sun Nov 24 14:56:00.410 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:199 435ms Sun Nov 24 14:56:00.464 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:56:00.465 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('5292599d0cf23a90f681e77a') } -->> { : MaxKey } Sun Nov 24 14:56:00.467 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d00cf23a90f681ef39') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292599d0cf23a90f681e77a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:00.489 [DataFileSync] flushing mmaps took 10534ms for 11 files Sun Nov 24 14:56:00.709 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:370766 185ms Sun Nov 24 14:56:01.077 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:504489 252ms Sun Nov 24 14:56:01.700 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:56:01.700 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('5292599d0cf23a90f681e77a') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259c60cf23a90f681ec8b') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292599d0cf23a90f681e77a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:166 reslen:37 11487ms Sun Nov 24 14:56:01.704 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:378785 189ms Sun Nov 24 14:56:01.783 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:01.788 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:01.998 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:356784 178ms Sun Nov 24 14:56:02.001 [conn7] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:91 reslen:3141 119ms Sun Nov 24 14:56:02.043 [conn7] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:56:02.044 [conn7] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom16/node41.clus.cci.emory.edu:27001", fromShard: "dicom1", toShard: "dicom16", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258f60cf23a90f681b682')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:56:02.432 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:637116 319ms Sun Nov 24 14:56:02.560 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259d218c2315fef764b1a Sun Nov 24 14:56:02.560 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:02-529259d218c2315fef764b1b", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322962560), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, from: "dicom1", to: "dicom16" } } Sun Nov 24 14:56:02.634 [conn7] moveChunk request accepted at version 7|3||529258b4c868398bb45e3755 Sun Nov 24 14:56:02.776 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:446513 223ms Sun Nov 24 14:56:02.780 [conn7] moveChunk number of documents: 477 Sun Nov 24 14:56:02.780 [conn7] starting new replica set monitor for replica set dicom16 with seed of node41.clus.cci.emory.edu:27001 Sun Nov 24 14:56:02.781 [conn7] successfully connected to seed node41.clus.cci.emory.edu:27001 for replica set dicom16 Sun Nov 24 14:56:02.782 [conn7] changing hosts to { 0: "node41.clus.cci.emory.edu:27001" } from dicom16/ Sun Nov 24 14:56:02.782 [conn7] trying to add new host node41.clus.cci.emory.edu:27001 to replica set dicom16 Sun Nov 24 14:56:02.782 [conn7] successfully connected to new host node41.clus.cci.emory.edu:27001 in replica set dicom16 Sun Nov 24 14:56:02.784 [conn7] Primary for replica set dicom16 changed to node41.clus.cci.emory.edu:27001 Sun Nov 24 14:56:02.785 [conn7] replica set monitor for replica set dicom16 started, address is dicom16/node41.clus.cci.emory.edu:27001 Sun Nov 24 14:56:02.792 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:02.796 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:02.802 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:02.812 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:02.830 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:02.864 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:02.930 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:03.060 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:03.137 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:260 242ms Sun Nov 24 14:56:03.137 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:484441 242ms Sun Nov 24 14:56:03.223 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:03.229 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:03.317 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:03.482 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:453947 227ms Sun Nov 24 14:56:03.830 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:03.898 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:601731 301ms Sun Nov 24 14:56:04.188 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:354231 177ms Sun Nov 24 14:56:04.458 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:312044 156ms Sun Nov 24 14:56:04.486 [conn3659] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33400 reslen:16519271 222ms Sun Nov 24 14:56:04.810 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:04.811 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:04.811 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:04.812 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:04.813 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d40cf23a90f681f324') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:04.814 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d40cf23a90f681f324') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:04.855 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 187, clonedBytes: 16551350, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:05.353 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:411031 205ms Sun Nov 24 14:56:05.626 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:320568 160ms Sun Nov 24 14:56:05.880 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 250, clonedBytes: 33070328, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:05.929 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:376689 188ms Sun Nov 24 14:56:06.296 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:500338 250ms Sun Nov 24 14:56:06.309 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:06.310 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:06.312 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d60cf23a90f681f497') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:06.313 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:06.332 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:06.334 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d60cf23a90f681f497') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:06.349 [conn3659] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33373 reslen:16561647 182ms Sun Nov 24 14:56:06.552 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:288397 144ms Sun Nov 24 14:56:06.878 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:421135 210ms Sun Nov 24 14:56:06.905 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 313, clonedBytes: 49589306, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:07.219 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:449294 225ms Sun Nov 24 14:56:07.219 [conn3643] command admin.$cmd command: { splitVector: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258de0cf23a90f681aff5') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 locks(micros) r:121 reslen:53 225ms Sun Nov 24 14:56:07.280 [conn3659] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:47930 reslen:16526323 243ms Sun Nov 24 14:56:07.482 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:302154 151ms Sun Nov 24 14:56:07.566 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:07.566 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:07.567 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:07.568 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:07.569 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d70cf23a90f681f5e5') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:07.570 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d70cf23a90f681f5e5') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:07.931 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 466, clonedBytes: 82676582, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:08.290 [conn3659] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:7762 reslen:2884351 403ms Sun Nov 24 14:56:08.360 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:08.360 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:08.361 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:08.361 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:08.363 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d80cf23a90f681f61f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:08.364 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d80cf23a90f681f61f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:08.575 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:371888 186ms Sun Nov 24 14:56:08.957 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 477, clonedBytes: 85560848, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:09.228 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:489497 245ms Sun Nov 24 14:56:09.230 [conn3659] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:51 162ms Sun Nov 24 14:56:09.594 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:499207 249ms Sun Nov 24 14:56:09.596 [conn3659] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:51 259ms Sun Nov 24 14:56:09.711 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:09.712 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:09.712 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:09.713 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:09.715 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d90cf23a90f681f796') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:09.716 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259d90cf23a90f681f796') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:09.947 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:247 229ms Sun Nov 24 14:56:09.947 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:196 228ms Sun Nov 24 14:56:09.981 [conn7] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 477, clonedBytes: 85560848, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:09.984 [conn7] moveChunk setting version to: 8|0||529258b4c868398bb45e3755 Sun Nov 24 14:56:09.990 [conn7] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 477, clonedBytes: 85560848, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:56:09.990 [conn7] moveChunk updating self version to: 8|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529258fe0cf23a90f681b98f') } -> { files_id: ObjectId('529259100cf23a90f681bc87') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:56:09.991 [conn3642] waiting till out of critical section Sun Nov 24 14:56:11.107 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:11-529259db18c2315fef764b1c", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322971107), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, from: "dicom1", to: "dicom16" } } Sun Nov 24 14:56:11.107 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 7000|3, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:249 reslen:325 1115ms Sun Nov 24 14:56:11.244 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:212532 106ms Sun Nov 24 14:56:11.293 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:11.294 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:11.295 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:11.296 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259db0cf23a90f681f7e1') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:11.296 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:11.298 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259db0cf23a90f681f7e1') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:11.610 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:497947 249ms Sun Nov 24 14:56:12.108 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:758062 379ms Sun Nov 24 14:56:12.248 [conn7] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:56:12.248 [LockPinger] trying to delete 1 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:56:12.529 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:608039 304ms Sun Nov 24 14:56:12.529 [conn7] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:56:12.529 [conn7] forking for cleanup of chunk data Sun Nov 24 14:56:12.529 [conn7] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:56:12.529 [cleanupOldData-529259dc18c2315fef764b1d] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529258f60cf23a90f681b682') } -> { files_id: ObjectId('529258fe0cf23a90f681b98f') }, # cursors remaining: 0 Sun Nov 24 14:56:12.531 [conn7] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:56:12.550 [cleanupOldData-529259dc18c2315fef764b1d] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529258f60cf23a90f681b682') } -> { files_id: ObjectId('529258fe0cf23a90f681b98f') } Sun Nov 24 14:56:12.550 [cleanupOldData-529259dc18c2315fef764b1d] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529258f60cf23a90f681b682') } -> { files_id: ObjectId('529258fe0cf23a90f681b98f') } Sun Nov 24 14:56:12.793 [LockPinger] handled late remove of old distributed lock with ts 529259c618c2315fef764b16 Sun Nov 24 14:56:12.793 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:56:12.793 [conn7] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:12-529259dc18c2315fef764b1e", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32201", time: new Date(1385322972793), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, step1 of 6: 1, step2 of 6: 589, step3 of 6: 154, step4 of 6: 7193, step5 of 6: 2547, step6 of 6: 0 } } Sun Nov 24 14:56:12.831 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:375236 188ms Sun Nov 24 14:56:12.850 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:12.851 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:12.853 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:12.854 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:12.855 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259dc0cf23a90f681f913') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:12.856 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259dc0cf23a90f681f913') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:12.884 [cleanupOldData-529259dc18c2315fef764b1d] moveChunk deleted 477 documents for dicomdb.fs.chunks from { files_id: ObjectId('529258f60cf23a90f681b682') } -> { files_id: ObjectId('529258fe0cf23a90f681b98f') } Sun Nov 24 14:56:13.082 [conn7] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom16/node41.clus.cci.emory.edu:27001", fromShard: "dicom1", toShard: "dicom16", min: { files_id: ObjectId('529258f60cf23a90f681b682') }, max: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258f60cf23a90f681b682')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:93 r:1318 w:46 reslen:37 11038ms Sun Nov 24 14:56:13.082 [conn3643] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529259d218c2315fef764b1a') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn3643:179229628", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322972855), why: "split-{ files_id: ObjectId('529259c60cf23a90f681ec8b') }", ts: ObjectId('529259dc18c2315fef764b1f') } } gle1: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322972000|2, connectionId: 29, waited: 30, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322972000|1, connectionId: 62, waited: 32, err: null, ok: 1.0 } Sun Nov 24 14:56:13.082 [conn10] warning: distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908 did not propagate properly. :: caused by :: 8017 update not consistent ns: config.locks query: { _id: "dicomdb.fs.chunks", state: 0, ts: ObjectId('529259d218c2315fef764b1a') } update: { $set: { state: 1, who: "node40.clus.cci.emory.edu:27001:1385321687:2071045908:conn10:1345868337", process: "node40.clus.cci.emory.edu:27001:1385321687:2071045908", when: new Date(1385322972856), why: "split-{ files_id: ObjectId('529259c60cf23a90f681ec8b') }", ts: ObjectId('529259dc18c2315fef764b20') } } gle1: { updatedExisting: false, n: 0, lastOp: Timestamp 1385322962000|2, connectionId: 42, waited: 29, err: null, ok: 1.0 } gle2: { updatedExisting: true, n: 1, lastOp: Timestamp 1385322972000|4, connectionId: 74, waited: 32, err: null, ok: 1.0 } Sun Nov 24 14:56:13.084 [conn3643] lock update lost, lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' not propagated. Sun Nov 24 14:56:13.084 [conn3643] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:56:13.084 [conn3643] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259dc0cf23a90f681f913') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:381 228ms Sun Nov 24 14:56:13.096 [conn10] lock update won, completing lock propagation for 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' Sun Nov 24 14:56:13.233 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:119 109ms Sun Nov 24 14:56:17.143 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:249 3796ms Sun Nov 24 14:56:17.555 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:304 299ms Sun Nov 24 14:56:18.210 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:350152 175ms Sun Nov 24 14:56:18.315 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259dc18c2315fef764b20 Sun Nov 24 14:56:18.317 [conn10] splitChunk accepted at version 8|1||529258b4c868398bb45e3755 Sun Nov 24 14:56:18.585 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:522200 261ms Sun Nov 24 14:56:18.623 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:18.624 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259c60cf23a90f681ec8b') } -->> { : MaxKey } Sun Nov 24 14:56:18.626 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259e20cf23a90f681fa92') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:18.948 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:294 251ms Sun Nov 24 14:56:19.287 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:455045 227ms Sun Nov 24 14:56:19.292 [conn10] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:19-529259e318c2315fef764b21", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:32381", time: new Date(1385322979292), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, lastmod: Timestamp 7000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: ObjectId('529259dc0cf23a90f681f913') }, lastmod: Timestamp 8000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, lastmod: Timestamp 8000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:56:19.628 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:457275 229ms Sun Nov 24 14:56:19.905 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:330349 165ms Sun Nov 24 14:56:20.030 [conn10] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:56:20.030 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259c60cf23a90f681ec8b') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259dc0cf23a90f681f913') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259c60cf23a90f681ec8b')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:176 reslen:37 7174ms Sun Nov 24 14:56:20.162 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:287894 144ms Sun Nov 24 14:56:20.162 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:54 reslen:3141 130ms Sun Nov 24 14:56:20.199 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:56:20.201 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom17/node41.clus.cci.emory.edu:27002", fromShard: "dicom1", toShard: "dicom17", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:56:20.236 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:20.242 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:20.554 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:554444 277ms Sun Nov 24 14:56:20.883 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:424466 212ms Sun Nov 24 14:56:21.028 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259e418c2315fef764b22 Sun Nov 24 14:56:21.028 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:21-529259e518c2315fef764b23", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322981028), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, from: "dicom1", to: "dicom17" } } Sun Nov 24 14:56:21.325 [conn15] moveChunk request accepted at version 8|3||529258b4c868398bb45e3755 Sun Nov 24 14:56:21.330 [conn15] moveChunk number of documents: 456 Sun Nov 24 14:56:21.330 [conn15] starting new replica set monitor for replica set dicom17 with seed of node41.clus.cci.emory.edu:27002 Sun Nov 24 14:56:21.331 [conn15] successfully connected to seed node41.clus.cci.emory.edu:27002 for replica set dicom17 Sun Nov 24 14:56:21.332 [conn15] changing hosts to { 0: "node41.clus.cci.emory.edu:27002" } from dicom17/ Sun Nov 24 14:56:21.332 [conn15] trying to add new host node41.clus.cci.emory.edu:27002 to replica set dicom17 Sun Nov 24 14:56:21.333 [conn15] successfully connected to new host node41.clus.cci.emory.edu:27002 in replica set dicom17 Sun Nov 24 14:56:21.334 [conn15] Primary for replica set dicom17 changed to node41.clus.cci.emory.edu:27002 Sun Nov 24 14:56:21.335 [conn15] replica set monitor for replica set dicom17 started, address is dicom17/node41.clus.cci.emory.edu:27002 Sun Nov 24 14:56:21.339 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.342 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.347 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.356 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.374 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.408 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.473 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.585 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:210 217ms Sun Nov 24 14:56:21.585 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:217572 217ms Sun Nov 24 14:56:21.604 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.681 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:21.682 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:21.861 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:21.951 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:272 249ms Sun Nov 24 14:56:21.952 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:499211 250ms Sun Nov 24 14:56:22.257 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:202 188ms Sun Nov 24 14:56:22.257 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:298 187ms Sun Nov 24 14:56:22.375 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:22.576 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:406790 203ms Sun Nov 24 14:56:22.832 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:288407 144ms Sun Nov 24 14:56:23.116 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:385 167ms Sun Nov 24 14:56:23.116 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:334131 167ms Sun Nov 24 14:56:23.183 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:23.187 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:23.402 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 214, clonedBytes: 16516120, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:23.527 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:590108 295ms Sun Nov 24 14:56:23.845 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:408079 204ms Sun Nov 24 14:56:24.435 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 277, clonedBytes: 33035098, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:24.528 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:353165 177ms Sun Nov 24 14:56:24.884 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:24.888 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:25.168 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:371442 185ms Sun Nov 24 14:56:25.223 [conn3663] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:32413 reslen:16519271 162ms Sun Nov 24 14:56:25.437 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:310481 155ms Sun Nov 24 14:56:25.461 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 340, clonedBytes: 49554076, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:25.853 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:601462 301ms Sun Nov 24 14:56:26.460 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:263 494ms Sun Nov 24 14:56:26.461 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:929655 465ms Sun Nov 24 14:56:26.484 [conn3663] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34215 reslen:13897171 522ms Sun Nov 24 14:56:26.488 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 403, clonedBytes: 66073054, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:26.614 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:26.615 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:26.615 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:26.616 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:26.617 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259ea0cf23a90f6820202') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:26.617 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259ea0cf23a90f6820202') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:26.745 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:232 124ms Sun Nov 24 14:56:26.745 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:251 123ms Sun Nov 24 14:56:27.117 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:509394 255ms Sun Nov 24 14:56:27.410 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:355836 178ms Sun Nov 24 14:56:27.514 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 456, clonedBytes: 79969972, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:27.524 [conn15] moveChunk setting version to: 9|0||529258b4c868398bb45e3755 Sun Nov 24 14:56:27.746 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:275 221ms Sun Nov 24 14:56:27.746 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:194 219ms Sun Nov 24 14:56:27.746 [conn3642] waiting till out of critical section Sun Nov 24 14:56:27.746 [conn3663] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:51 209ms Sun Nov 24 14:56:27.750 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 456, clonedBytes: 79969972, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:56:27.750 [conn15] moveChunk updating self version to: 9|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529259100cf23a90f681bc87') } -> { files_id: ObjectId('529259180cf23a90f681c05a') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:56:30.033 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:30-529259ee18c2315fef764b24", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322990033), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, from: "dicom1", to: "dicom17" } } Sun Nov 24 14:56:30.033 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 8000|3, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:69 reslen:325 2505ms Sun Nov 24 14:56:30.126 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:30.130 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:30.147 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:30.147 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:30.149 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259ee0cf23a90f682034f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:30.149 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259ee0cf23a90f682034f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:30.590 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:343 437ms Sun Nov 24 14:56:30.590 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:297 436ms Sun Nov 24 14:56:31.396 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:829973 415ms Sun Nov 24 14:56:31.743 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:360 231ms Sun Nov 24 14:56:31.743 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:463107 231ms Sun Nov 24 14:56:32.029 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:56:32.066 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:414813 207ms Sun Nov 24 14:56:32.066 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:56:32.066 [conn15] forking for cleanup of chunk data Sun Nov 24 14:56:32.066 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:56:32.066 [cleanupOldData-529259f018c2315fef764b25] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529258fe0cf23a90f681b98f') } -> { files_id: ObjectId('529259100cf23a90f681bc87') }, # cursors remaining: 0 Sun Nov 24 14:56:32.068 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:56:32.088 [cleanupOldData-529259f018c2315fef764b25] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529258fe0cf23a90f681b98f') } -> { files_id: ObjectId('529259100cf23a90f681bc87') } Sun Nov 24 14:56:32.088 [cleanupOldData-529259f018c2315fef764b25] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529258fe0cf23a90f681b98f') } -> { files_id: ObjectId('529259100cf23a90f681bc87') } Sun Nov 24 14:56:32.470 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:570725 285ms Sun Nov 24 14:56:32.495 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:32.496 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:32.497 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:32.498 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259f00cf23a90f68204be') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:32.499 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:32.501 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259f00cf23a90f68204be') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:32.754 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:336189 168ms Sun Nov 24 14:56:32.763 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:56:32.763 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:32-529259f018c2315fef764b26", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322992763), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, step1 of 6: 1, step2 of 6: 1124, step3 of 6: 11, step4 of 6: 6178, step5 of 6: 4551, step6 of 6: 0 } } Sun Nov 24 14:56:32.766 [cleanupOldData-529259f018c2315fef764b25] moveChunk deleted 456 documents for dicomdb.fs.chunks from { files_id: ObjectId('529258fe0cf23a90f681b98f') } -> { files_id: ObjectId('529259100cf23a90f681bc87') } Sun Nov 24 14:56:33.095 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:446818 223ms Sun Nov 24 14:56:33.195 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom17/node41.clus.cci.emory.edu:27002", fromShard: "dicom1", toShard: "dicom17", min: { files_id: ObjectId('529258fe0cf23a90f681b98f') }, max: { files_id: ObjectId('529259100cf23a90f681bc87') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529258fe0cf23a90f681b98f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 numYields: 2 locks(micros) W:193 r:2445 w:44 reslen:37 12995ms Sun Nov 24 14:56:33.739 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:292 527ms Sun Nov 24 14:56:33.740 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1054777 527ms Sun Nov 24 14:56:33.849 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:33.850 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:33.851 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:33.851 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:33.854 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259f10cf23a90f68205fb') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:33.854 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259f10cf23a90f68205fb') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:34.135 [conn7] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:56:34.135 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:56:34.135 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259f10cf23a90f68205fb') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:382 280ms Sun Nov 24 14:56:34.180 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:34.181 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259dc0cf23a90f681f913') } -->> { : MaxKey } Sun Nov 24 14:56:34.183 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259f20cf23a90f6820602') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:34.368 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259f118c2315fef764b27 Sun Nov 24 14:56:34.370 [conn3641] splitChunk accepted at version 9|1||529258b4c868398bb45e3755 Sun Nov 24 14:56:34.444 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:338335 169ms Sun Nov 24 14:56:34.586 [conn3641] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:34-529259f218c2315fef764b29", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:34240", time: new Date(1385322994586), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, lastmod: Timestamp 8000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: ObjectId('529259f10cf23a90f68205fb') }, lastmod: Timestamp 9000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, lastmod: Timestamp 9000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:56:34.858 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:598271 299ms Sun Nov 24 14:56:34.984 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:56:34.984 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259dc0cf23a90f681f913') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259f10cf23a90f68205fb') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259dc0cf23a90f681f913')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:98 reslen:37 1130ms Sun Nov 24 14:56:35.366 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:35.372 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:35.794 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:264 366ms Sun Nov 24 14:56:35.794 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:208 365ms Sun Nov 24 14:56:35.908 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:56:35.910 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom18/node41.clus.cci.emory.edu:27003", fromShard: "dicom1", toShard: "dicom18", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:56:36.011 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259f318c2315fef764b2a Sun Nov 24 14:56:36.011 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:36-529259f418c2315fef764b2b", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385322996011), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, from: "dicom1", to: "dicom18" } } Sun Nov 24 14:56:36.447 [journal] old journal file will be removed: /shared/mongodb/data/db/dicom/dicom1/journal/j._3 Sun Nov 24 14:56:36.585 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1339335 670ms Sun Nov 24 14:56:36.586 [conn15] moveChunk request accepted at version 9|3||529258b4c868398bb45e3755 Sun Nov 24 14:56:36.590 [conn15] moveChunk number of documents: 404 Sun Nov 24 14:56:36.590 [conn15] starting new replica set monitor for replica set dicom18 with seed of node41.clus.cci.emory.edu:27003 Sun Nov 24 14:56:36.591 [conn15] successfully connected to seed node41.clus.cci.emory.edu:27003 for replica set dicom18 Sun Nov 24 14:56:36.591 [conn15] changing hosts to { 0: "node41.clus.cci.emory.edu:27003" } from dicom18/ Sun Nov 24 14:56:36.591 [conn15] trying to add new host node41.clus.cci.emory.edu:27003 to replica set dicom18 Sun Nov 24 14:56:36.592 [conn15] successfully connected to new host node41.clus.cci.emory.edu:27003 in replica set dicom18 Sun Nov 24 14:56:36.594 [conn15] Primary for replica set dicom18 changed to node41.clus.cci.emory.edu:27003 Sun Nov 24 14:56:36.594 [conn15] replica set monitor for replica set dicom18 started, address is dicom18/node41.clus.cci.emory.edu:27003 Sun Nov 24 14:56:36.598 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:36.602 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:36.609 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:36.619 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:36.637 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:36.672 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:36.738 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:36.869 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:37.127 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:37.135 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:862446 431ms Sun Nov 24 14:56:37.208 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:37.213 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:37.614 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:723973 362ms Sun Nov 24 14:56:37.642 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:38.161 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:835600 418ms Sun Nov 24 14:56:38.222 [conn3667] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33458 reslen:16688736 410ms Sun Nov 24 14:56:38.667 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 123, clonedBytes: 16688180, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:38.765 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:971649 486ms Sun Nov 24 14:56:38.820 [conn3667] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33007 reslen:16751663 339ms Sun Nov 24 14:56:39.103 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:39.107 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:39.357 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:385586 193ms Sun Nov 24 14:56:39.656 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:360224 180ms Sun Nov 24 14:56:39.691 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 189, clonedBytes: 33439538, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:39.713 [conn3667] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:32585 reslen:16692371 155ms Sun Nov 24 14:56:39.911 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:285655 143ms Sun Nov 24 14:56:40.311 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:40.315 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:40.523 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:356895 178ms Sun Nov 24 14:56:40.582 [conn3667] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:32439 reslen:16691187 182ms Sun Nov 24 14:56:40.726 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 272, clonedBytes: 50131536, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:40.837 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:398973 199ms Sun Nov 24 14:56:41.175 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:445060 222ms Sun Nov 24 14:56:41.201 [conn3667] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:14140 reslen:7316601 148ms Sun Nov 24 14:56:41.503 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:428669 214ms Sun Nov 24 14:56:41.752 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 404, clonedBytes: 74138714, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:42.300 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1363840 682ms Sun Nov 24 14:56:42.302 [conn3667] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:55 reslen:51 625ms Sun Nov 24 14:56:42.310 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:42.311 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:42.630 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:430773 215ms Sun Nov 24 14:56:42.633 [conn3667] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:46 reslen:51 220ms Sun Nov 24 14:56:42.859 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 404, clonedBytes: 74138714, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:42.980 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:472588 236ms Sun Nov 24 14:56:42.982 [conn15] moveChunk setting version to: 10|0||529258b4c868398bb45e3755 Sun Nov 24 14:56:42.982 [conn3667] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:45 reslen:51 239ms Sun Nov 24 14:56:42.987 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 404, clonedBytes: 74138714, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:56:42.988 [conn15] moveChunk updating self version to: 10|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529259180cf23a90f681c05a') } -> { files_id: ObjectId('5292592d0cf23a90f681c5e0') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:56:42.989 [conn3642] waiting till out of critical section Sun Nov 24 14:56:44.450 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:44-529259fc18c2315fef764b2c", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385323004449), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, from: "dicom1", to: "dicom18" } } Sun Nov 24 14:56:44.450 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 9000|3, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:146 reslen:325 1460ms Sun Nov 24 14:56:44.693 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:241 190ms Sun Nov 24 14:56:44.694 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:381686 191ms Sun Nov 24 14:56:44.734 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:44.735 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:44.736 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:44.737 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259fc0cf23a90f6820c98') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259f10cf23a90f68205fb')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:44.737 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:44.740 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259fc0cf23a90f6820c98') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259f10cf23a90f68205fb')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:45.074 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:229 264ms Sun Nov 24 14:56:45.075 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:529839 265ms Sun Nov 24 14:56:45.220 [LockPinger] trying to delete 1 old lock entries for process node40.clus.cci.emory.edu:27001:1385321687:2071045908 Sun Nov 24 14:56:45.220 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:56:45.471 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:559518 280ms Sun Nov 24 14:56:45.471 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:56:45.471 [conn15] forking for cleanup of chunk data Sun Nov 24 14:56:45.472 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:56:45.472 [cleanupOldData-529259fd18c2315fef764b2d] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529259100cf23a90f681bc87') } -> { files_id: ObjectId('529259180cf23a90f681c05a') }, # cursors remaining: 0 Sun Nov 24 14:56:45.473 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:56:45.493 [cleanupOldData-529259fd18c2315fef764b2d] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529259100cf23a90f681bc87') } -> { files_id: ObjectId('529259180cf23a90f681c05a') } Sun Nov 24 14:56:45.493 [cleanupOldData-529259fd18c2315fef764b2d] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529259100cf23a90f681bc87') } -> { files_id: ObjectId('529259180cf23a90f681c05a') } Sun Nov 24 14:56:45.792 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:409527 205ms Sun Nov 24 14:56:45.851 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:56:45.851 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:45-529259fd18c2315fef764b2e", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385323005851), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, step1 of 6: 1, step2 of 6: 676, step3 of 6: 10, step4 of 6: 6263, step5 of 6: 2611, step6 of 6: 0 } } Sun Nov 24 14:56:45.852 [LockPinger] handled late remove of old distributed lock with ts 529259dc18c2315fef764b1f Sun Nov 24 14:56:45.878 [cleanupOldData-529259fd18c2315fef764b2d] moveChunk deleted 404 documents for dicomdb.fs.chunks from { files_id: ObjectId('529259100cf23a90f681bc87') } -> { files_id: ObjectId('529259180cf23a90f681c05a') } Sun Nov 24 14:56:45.957 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom18/node41.clus.cci.emory.edu:27003", fromShard: "dicom1", toShard: "dicom18", min: { files_id: ObjectId('529259100cf23a90f681bc87') }, max: { files_id: ObjectId('529259180cf23a90f681c05a') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259100cf23a90f681bc87')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:256 r:1674 w:46 reslen:37 10048ms Sun Nov 24 14:56:46.156 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:242 247ms Sun Nov 24 14:56:46.156 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:295 247ms Sun Nov 24 14:56:46.196 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:46.197 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:46.199 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259fe0cf23a90f6820ddf') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259f10cf23a90f68205fb')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:46.202 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:46.204 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259f10cf23a90f68205fb') } -->> { : MaxKey } Sun Nov 24 14:56:46.205 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259fe0cf23a90f6820ddf') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259f10cf23a90f68205fb')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:56:46.645 [conn7] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:56:46.645 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:56:46.645 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259fe0cf23a90f6820ddf') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259f10cf23a90f68205fb')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:382 440ms Sun Nov 24 14:56:46.720 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 529259fe18c2315fef764b2f Sun Nov 24 14:56:46.723 [conn3641] splitChunk accepted at version 10|1||529258b4c868398bb45e3755 Sun Nov 24 14:56:46.795 [conn3641] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:46-529259fe18c2315fef764b31", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:34240", time: new Date(1385323006795), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, lastmod: Timestamp 9000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, lastmod: Timestamp 10000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, lastmod: Timestamp 10000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:56:46.912 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:345258 173ms Sun Nov 24 14:56:47.524 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:469445 235ms Sun Nov 24 14:56:47.807 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:342265 171ms Sun Nov 24 14:56:47.879 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:56:47.879 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259f10cf23a90f68205fb') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('529259fe0cf23a90f6820ddf') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259f10cf23a90f68205fb')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:166 reslen:37 1680ms Sun Nov 24 14:56:48.067 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:294641 147ms Sun Nov 24 14:56:48.106 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:56:48.107 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom19/node41.clus.cci.emory.edu:27004", fromShard: "dicom1", toShard: "dicom19", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259180cf23a90f681c05a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:56:48.110 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:48.115 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:49.518 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:2671758 1336ms Sun Nov 24 14:56:49.863 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:459081 229ms Sun Nov 24 14:56:50.211 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 52925a0018c2315fef764b32 Sun Nov 24 14:56:50.211 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:56:50-52925a0218c2315fef764b33", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385323010211), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, from: "dicom1", to: "dicom19" } } Sun Nov 24 14:56:50.465 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:971505 486ms Sun Nov 24 14:56:51.173 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:241 592ms Sun Nov 24 14:56:51.173 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:395 592ms Sun Nov 24 14:56:51.271 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:51.272 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:52.341 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:290 1051ms Sun Nov 24 14:56:52.341 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:377 1050ms Sun Nov 24 14:56:53.466 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:2017721 1009ms Sun Nov 24 14:56:54.004 [conn15] moveChunk request accepted at version 10|3||529258b4c868398bb45e3755 Sun Nov 24 14:56:54.660 [conn15] moveChunk number of documents: 677 Sun Nov 24 14:56:54.661 [conn15] starting new replica set monitor for replica set dicom19 with seed of node41.clus.cci.emory.edu:27004 Sun Nov 24 14:56:54.662 [conn15] successfully connected to seed node41.clus.cci.emory.edu:27004 for replica set dicom19 Sun Nov 24 14:56:54.662 [conn15] changing hosts to { 0: "node41.clus.cci.emory.edu:27004" } from dicom19/ Sun Nov 24 14:56:54.664 [conn15] trying to add new host node41.clus.cci.emory.edu:27004 to replica set dicom19 Sun Nov 24 14:56:54.666 [conn15] successfully connected to new host node41.clus.cci.emory.edu:27004 in replica set dicom19 Sun Nov 24 14:56:54.668 [conn15] Primary for replica set dicom19 changed to node41.clus.cci.emory.edu:27004 Sun Nov 24 14:56:54.669 [conn15] replica set monitor for replica set dicom19 started, address is dicom19/node41.clus.cci.emory.edu:27004 Sun Nov 24 14:56:54.673 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:54.676 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:54.681 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:54.690 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:54.707 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:54.740 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:54.805 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:54.938 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:55.195 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:55.709 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:56.130 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1970574 985ms Sun Nov 24 14:56:56.184 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:56.185 [conn3671] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33674 reslen:16675171 366ms Sun Nov 24 14:56:56.188 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:56.735 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 82, clonedBytes: 16674802, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:57.126 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:313 877ms Sun Nov 24 14:56:57.126 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:439 877ms Sun Nov 24 14:56:57.761 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 166, clonedBytes: 33398914, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:58.299 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:324 1053ms Sun Nov 24 14:56:58.299 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:384 1052ms Sun Nov 24 14:56:58.787 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:56:59.073 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1314846 657ms Sun Nov 24 14:56:59.430 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:282 241ms Sun Nov 24 14:56:59.430 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:352 241ms Sun Nov 24 14:56:59.487 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:59.491 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:56:59.821 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:550094 275ms Sun Nov 24 14:56:59.935 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:00.287 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:337 345ms Sun Nov 24 14:57:00.287 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:377 346ms Sun Nov 24 14:57:00.822 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:823622 412ms Sun Nov 24 14:57:00.961 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:01.341 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:01.348 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:01.734 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:601557 301ms Sun Nov 24 14:57:01.987 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:02.602 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:321 308ms Sun Nov 24 14:57:02.602 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:291 308ms Sun Nov 24 14:57:02.924 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:377 204ms Sun Nov 24 14:57:02.926 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:207220 206ms Sun Nov 24 14:57:02.967 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:02.967 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:02.968 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a0e0cf23a90f68215e2') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:02.972 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:02.973 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:02.975 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a0e0cf23a90f68215e2') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:03.013 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:03.299 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:514144 257ms Sun Nov 24 14:57:03.722 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:420 305ms Sun Nov 24 14:57:03.723 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:611383 306ms Sun Nov 24 14:57:04.039 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:04.163 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:649496 325ms Sun Nov 24 14:57:04.711 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:290 429ms Sun Nov 24 14:57:04.711 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:374 429ms Sun Nov 24 14:57:05.088 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:05.165 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:257 342ms Sun Nov 24 14:57:05.165 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:278 342ms Sun Nov 24 14:57:05.182 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:05.183 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:05.185 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a110cf23a90f6821756') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:05.188 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:05.189 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:05.191 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a110cf23a90f6821756') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:05.625 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:300 341ms Sun Nov 24 14:57:05.625 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:417 341ms Sun Nov 24 14:57:05.982 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:323 238ms Sun Nov 24 14:57:05.982 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:447 238ms Sun Nov 24 14:57:06.114 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:06.956 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:06.958 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:06.960 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a120cf23a90f68218cd') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:06.961 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:06.963 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:06.965 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a120cf23a90f68218cd') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:07.140 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:07.350 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:308 295ms Sun Nov 24 14:57:07.350 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:459 295ms Sun Nov 24 14:57:07.365 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:07.366 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:07.366 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:07.367 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:07.368 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a130cf23a90f682190d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:07.368 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a130cf23a90f682190d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:07.755 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:580977 291ms Sun Nov 24 14:57:08.166 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:08.264 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:787443 394ms Sun Nov 24 14:57:08.737 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:712661 356ms Sun Nov 24 14:57:09.192 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:09.476 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1239026 620ms Sun Nov 24 14:57:09.519 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:09.520 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:09.522 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a150cf23a90f6821a7f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:09.523 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:09.524 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:09.526 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a150cf23a90f6821a7f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:09.945 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:701641 351ms Sun Nov 24 14:57:10.248 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:10.562 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:282 501ms Sun Nov 24 14:57:10.562 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:333 500ms Sun Nov 24 14:57:11.269 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1182051 591ms Sun Nov 24 14:57:11.275 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 222, clonedBytes: 43819720, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:12.091 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.092 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.094 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a180cf23a90f6821bee') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:12.097 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.098 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.100 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a180cf23a90f6821bee') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:12.301 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 223, clonedBytes: 44081926, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:12.633 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:979547 490ms Sun Nov 24 14:57:12.691 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.692 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.692 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.693 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:12.694 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a180cf23a90f6821c2e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:12.695 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a180cf23a90f6821c2e') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:13.327 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 223, clonedBytes: 44081926, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:13.713 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1157500 579ms Sun Nov 24 14:57:14.353 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 223, clonedBytes: 44081926, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:14.475 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:265 647ms Sun Nov 24 14:57:14.475 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:258 647ms Sun Nov 24 14:57:14.904 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:299 310ms Sun Nov 24 14:57:14.904 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:434 310ms Sun Nov 24 14:57:15.015 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:15.016 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:15.016 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:15.017 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:15.018 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a1b0cf23a90f6821d99') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:15.019 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a1b0cf23a90f6821d99') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:15.379 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 311, clonedBytes: 60786130, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:15.529 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:333 509ms Sun Nov 24 14:57:15.530 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:180 507ms Sun Nov 24 14:57:16.230 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:347 580ms Sun Nov 24 14:57:16.230 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:428 580ms Sun Nov 24 14:57:16.405 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 341, clonedBytes: 66791264, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:16.937 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1179970 590ms Sun Nov 24 14:57:17.431 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 434, clonedBytes: 83364138, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:17.442 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:781573 391ms Sun Nov 24 14:57:18.021 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:454 458ms Sun Nov 24 14:57:18.021 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:264 458ms Sun Nov 24 14:57:18.041 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.042 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.043 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.043 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.045 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a1e0cf23a90f6821ed2') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:18.045 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a1e0cf23a90f6821ed2') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:18.119 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.120 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.120 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.121 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:18.123 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a1e0cf23a90f6821efa') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:18.123 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a1e0cf23a90f6821efa') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:18.456 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 529, clonedBytes: 99940340, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:18.600 [conn3671] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34513 reslen:16576655 319ms Sun Nov 24 14:57:19.126 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:289 471ms Sun Nov 24 14:57:19.126 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:942492 471ms Sun Nov 24 14:57:19.481 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 624, clonedBytes: 116516574, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:19.824 [conn3671] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:19184 reslen:9209671 342ms Sun Nov 24 14:57:20.385 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:236 471ms Sun Nov 24 14:57:20.385 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:942236 471ms Sun Nov 24 14:57:20.450 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:20.452 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:20.453 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:20.454 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:20.455 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a200cf23a90f6821fe0') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:20.456 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a200cf23a90f6821fe0') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:20.506 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "catchup", counts: { cloned: 677, clonedBytes: 125725992, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:20.965 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:927440 464ms Sun Nov 24 14:57:20.967 [conn3671] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:42 reslen:51 213ms Sun Nov 24 14:57:21.531 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 677, clonedBytes: 125725992, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:21.698 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:316 612ms Sun Nov 24 14:57:21.699 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1226097 613ms Sun Nov 24 14:57:21.699 [conn15] moveChunk setting version to: 11|0||529258b4c868398bb45e3755 Sun Nov 24 14:57:21.699 [conn3671] command admin.$cmd command: { _transferMods: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:54 reslen:51 623ms Sun Nov 24 14:57:21.700 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:1975 37234ms Sun Nov 24 14:57:21.732 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 677, clonedBytes: 125725992, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:57:21.732 [conn15] moveChunk updating self version to: 11|1||529258b4c868398bb45e3755 through { files_id: ObjectId('5292592d0cf23a90f681c5e0') } -> { files_id: ObjectId('529259620cf23a90f681d1ec') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:57:21.741 [conn3642] waiting till out of critical section Sun Nov 24 14:57:24.891 [DataFileSync] flushing mmaps took 34934ms for 11 files Sun Nov 24 14:57:25.201 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:57:25-52925a2518c2315fef764b34", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385323045201), what: "moveChunk.commit", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, from: "dicom1", to: "dicom19" } } Sun Nov 24 14:57:25.201 [conn3642] command admin.$cmd command: { setShardVersion: "dicomdb.fs.chunks", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", version: Timestamp 10000|3, versionEpoch: ObjectId('529258b4c868398bb45e3755'), serverID: ObjectId('528f8628c868398bb45de20e'), shard: "dicom1", shardHost: "dicom1/node40.clus.cci.emory.edu:27001" } ntoreturn:1 keyUpdates:0 numYields: 1 locks(micros) W:93 reslen:325 3459ms Sun Nov 24 14:57:25.352 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:228003 114ms Sun Nov 24 14:57:25.402 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:25.403 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:25.405 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a250cf23a90f6822069') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:25.405 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:25.406 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:25.408 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a250cf23a90f6822069') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:25.727 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:57:25.738 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:540513 270ms Sun Nov 24 14:57:25.738 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:57:25.739 [conn15] forking for cleanup of chunk data Sun Nov 24 14:57:25.739 [conn15] MigrateFromStatus::done About to acquire global write lock to exit critical section Sun Nov 24 14:57:25.739 [cleanupOldData-52925a2518c2315fef764b35] (start) waiting to cleanup dicomdb.fs.chunks from { files_id: ObjectId('529259180cf23a90f681c05a') } -> { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, # cursors remaining: 0 Sun Nov 24 14:57:25.741 [conn15] MigrateFromStatus::done Global lock acquired Sun Nov 24 14:57:25.760 [cleanupOldData-52925a2518c2315fef764b35] waiting to remove documents for dicomdb.fs.chunks from { files_id: ObjectId('529259180cf23a90f681c05a') } -> { files_id: ObjectId('5292592d0cf23a90f681c5e0') } Sun Nov 24 14:57:25.760 [cleanupOldData-52925a2518c2315fef764b35] moveChunk starting delete for: dicomdb.fs.chunks from { files_id: ObjectId('529259180cf23a90f681c05a') } -> { files_id: ObjectId('5292592d0cf23a90f681c5e0') } Sun Nov 24 14:57:25.923 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:57:25.923 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:57:25-52925a2518c2315fef764b36", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385323045923), what: "moveChunk.from", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, step1 of 6: 1, step2 of 6: 5897, step3 of 6: 666, step4 of 6: 26860, step5 of 6: 4207, step6 of 6: 0 } } Sun Nov 24 14:57:26.144 [conn15] command admin.$cmd command: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom19/node41.clus.cci.emory.edu:27004", fromShard: "dicom1", toShard: "dicom19", min: { files_id: ObjectId('529259180cf23a90f681c05a') }, max: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259180cf23a90f681c05a')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } ntoreturn:1 keyUpdates:0 locks(micros) W:238 r:2092 w:54 reslen:37 38038ms Sun Nov 24 14:57:26.161 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:303 299ms Sun Nov 24 14:57:26.161 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:295 298ms Sun Nov 24 14:57:26.566 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:572131 286ms Sun Nov 24 14:57:26.640 [conn7] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:26.641 [conn3641] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:26.643 [conn7] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:26.643 [conn3641] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('529259fe0cf23a90f6820ddf') } -->> { : MaxKey } Sun Nov 24 14:57:26.646 [conn3641] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a260cf23a90f682214f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:26.646 [conn7] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a260cf23a90f682214f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:26.793 [conn7] could not acquire lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' (another update won) Sun Nov 24 14:57:26.793 [conn7] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' was not acquired. Sun Nov 24 14:57:26.793 [conn7] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a260cf23a90f682214f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:382 147ms Sun Nov 24 14:57:26.946 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:304661 152ms Sun Nov 24 14:57:26.949 [cleanupOldData-52925a2518c2315fef764b35] moveChunk deleted 677 documents for dicomdb.fs.chunks from { files_id: ObjectId('529259180cf23a90f681c05a') } -> { files_id: ObjectId('5292592d0cf23a90f681c5e0') } Sun Nov 24 14:57:27.039 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 52925a2618c2315fef764b37 Sun Nov 24 14:57:27.041 [conn3641] splitChunk accepted at version 11|1||529258b4c868398bb45e3755 Sun Nov 24 14:57:27.075 [conn3641] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:57:27-52925a2718c2315fef764b39", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.159:34240", time: new Date(1385323047075), what: "split", ns: "dicomdb.fs.chunks", details: { before: { min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, lastmod: Timestamp 10000|3, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: ObjectId('52925a260cf23a90f682214f') }, lastmod: Timestamp 11000|2, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') }, right: { min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, lastmod: Timestamp 11000|3, lastmodEpoch: ObjectId('529258b4c868398bb45e3755') } } } Sun Nov 24 14:57:27.344 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:239 283ms Sun Nov 24 14:57:27.728 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:266155 266ms Sun Nov 24 14:57:27.815 [conn3641] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' unlocked. Sun Nov 24 14:57:27.815 [conn3641] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529259fe0cf23a90f6820ddf') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a260cf23a90f682214f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('529259fe0cf23a90f6820ddf')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 locks(micros) r:149 reslen:37 1169ms Sun Nov 24 14:57:27.853 [conn15] warning: not enough nodes in set to use secondaryThrottle: majority: 1 Sun Nov 24 14:57:27.854 [conn15] received moveChunk request: { moveChunk: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", to: "dicom2/node40.clus.cci.emory.edu:27002", fromShard: "dicom1", toShard: "dicom2", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, maxChunkSizeBytes: 268435456, shardId: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020", secondaryThrottle: true, waitForDelete: false } Sun Nov 24 14:57:28.003 [conn15] distributed lock 'dicomdb.fs.chunks/node40.clus.cci.emory.edu:27001:1385321687:2071045908' acquired, ts : 52925a2718c2315fef764b3a Sun Nov 24 14:57:28.003 [conn15] about to log metadata event: { _id: "node40.clus.cci.emory.edu-2013-11-24T19:57:28-52925a2818c2315fef764b3b", server: "node40.clus.cci.emory.edu", clientAddr: "192.168.1.160:48264", time: new Date(1385323048003), what: "moveChunk.start", ns: "dicomdb.fs.chunks", details: { min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, from: "dicom1", to: "dicom2" } } Sun Nov 24 14:57:28.127 [conn15] moveChunk request accepted at version 11|3||529258b4c868398bb45e3755 Sun Nov 24 14:57:28.141 [conn15] moveChunk number of documents: 1618 Sun Nov 24 14:57:28.141 [conn15] starting new replica set monitor for replica set dicom2 with seed of node40.clus.cci.emory.edu:27002 Sun Nov 24 14:57:28.143 [conn15] successfully connected to seed node40.clus.cci.emory.edu:27002 for replica set dicom2 Sun Nov 24 14:57:28.143 [conn15] changing hosts to { 0: "node40.clus.cci.emory.edu:27002" } from dicom2/ Sun Nov 24 14:57:28.143 [conn15] trying to add new host node40.clus.cci.emory.edu:27002 to replica set dicom2 Sun Nov 24 14:57:28.144 [conn15] successfully connected to new host node40.clus.cci.emory.edu:27002 in replica set dicom2 Sun Nov 24 14:57:28.146 [conn15] Primary for replica set dicom2 changed to node40.clus.cci.emory.edu:27002 Sun Nov 24 14:57:28.146 [conn15] replica set monitor for replica set dicom2 started, address is dicom2/node40.clus.cci.emory.edu:27002 Sun Nov 24 14:57:28.150 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.153 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.158 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:28.158 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.160 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:28.167 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.195 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.228 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.293 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.422 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:28.664 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:891341 446ms Sun Nov 24 14:57:28.679 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:29.192 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:29.233 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:895838 448ms Sun Nov 24 14:57:29.931 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:29.931 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:30.140 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:380186 190ms Sun Nov 24 14:57:30.207 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33819 reslen:16594501 135ms Sun Nov 24 14:57:30.217 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 81, clonedBytes: 16713138, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:30.730 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:950569 475ms Sun Nov 24 14:57:31.242 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 97, clonedBytes: 20388358, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:31.540 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:214 690ms Sun Nov 24 14:57:31.540 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:1379969 690ms Sun Nov 24 14:57:31.592 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33399 reslen:16573063 297ms Sun Nov 24 14:57:32.002 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:685240 343ms Sun Nov 24 14:57:32.054 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:32.055 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:32.267 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 166, clonedBytes: 33307258, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:32.504 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:768815 384ms Sun Nov 24 14:57:32.555 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:30588 reslen:16673518 153ms Sun Nov 24 14:57:32.971 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:351341 351ms Sun Nov 24 14:57:33.292 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 408, clonedBytes: 66552740, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:33.545 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:209 451ms Sun Nov 24 14:57:33.545 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:294 451ms Sun Nov 24 14:57:33.638 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:33.639 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:34.242 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34263 reslen:16717137 245ms Sun Nov 24 14:57:34.318 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 474, clonedBytes: 83246414, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:34.618 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:666780 333ms Sun Nov 24 14:57:35.448 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1414966 708ms Sun Nov 24 14:57:35.497 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 540, clonedBytes: 99963246, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:35.527 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34060 reslen:16573911 364ms Sun Nov 24 14:57:36.033 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:36.034 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:36.035 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:36.036 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:36.036 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a300cf23a90f68225b6') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:36.038 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a300cf23a90f68225b6') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:36.523 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 633, clonedBytes: 116536744, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:37.549 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 633, clonedBytes: 116536744, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:38.277 [conn17] command admin.$cmd command: { writebacklisten: ObjectId('528f975f583c9506c23d6230') } ntoreturn:1 keyUpdates:0 reslen:44 300038ms Sun Nov 24 14:57:38.373 [journal] old journal file will be removed: /shared/mongodb/data/db/dicom/dicom1/journal/j._4 Sun Nov 24 14:57:38.500 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:4801837 2401ms Sun Nov 24 14:57:38.566 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34987 reslen:16577583 2196ms Sun Nov 24 14:57:38.575 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 633, clonedBytes: 116536744, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:38.947 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:664970 333ms Sun Nov 24 14:57:39.004 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33213 reslen:16576407 321ms Sun Nov 24 14:57:39.462 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:333 394ms Sun Nov 24 14:57:39.462 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:359 394ms Sun Nov 24 14:57:39.600 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 822, clonedBytes: 149689896, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:39.875 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:183 295ms Sun Nov 24 14:57:39.875 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:383 295ms Sun Nov 24 14:57:39.908 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:39.909 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:39.910 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:39.911 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a330cf23a90f6822692') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:39.911 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:39.914 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a330cf23a90f6822692') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:40.247 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:503505 252ms Sun Nov 24 14:57:40.316 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34448 reslen:16575055 109ms Sun Nov 24 14:57:40.626 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 918, clonedBytes: 166270138, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:40.725 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:725780 363ms Sun Nov 24 14:57:41.503 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1322740 661ms Sun Nov 24 14:57:41.554 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:32735 reslen:16578071 245ms Sun Nov 24 14:57:41.652 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1011, clonedBytes: 182844780, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:41.996 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:753447 377ms Sun Nov 24 14:57:42.023 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.024 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.025 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.026 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.027 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a360cf23a90f6822772') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:42.030 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a360cf23a90f6822772') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:42.494 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.495 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.496 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.496 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:42.498 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a360cf23a90f68227a0') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:42.498 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a360cf23a90f68227a0') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:42.678 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1106, clonedBytes: 199422430, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:43.034 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:991749 496ms Sun Nov 24 14:57:43.104 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34672 reslen:16576955 506ms Sun Nov 24 14:57:43.704 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1106, clonedBytes: 199422430, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:44.469 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1250671 625ms Sun Nov 24 14:57:44.518 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:35200 reslen:16553493 689ms Sun Nov 24 14:57:44.730 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1201, clonedBytes: 215998964, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:45.236 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:202 646ms Sun Nov 24 14:57:45.236 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:253 646ms Sun Nov 24 14:57:45.273 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:45.274 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:45.275 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:45.276 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:45.276 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a390cf23a90f6822885') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:45.279 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a390cf23a90f6822885') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:45.756 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1291, clonedBytes: 232552056, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:45.822 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:921782 461ms Sun Nov 24 14:57:46.541 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1200867 600ms Sun Nov 24 14:57:46.595 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:32906 reslen:16669929 402ms Sun Nov 24 14:57:46.782 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1384, clonedBytes: 249292428, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:47.126 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:921915 461ms Sun Nov 24 14:57:47.191 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.192 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.193 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.193 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.195 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a3b0cf23a90f6822965') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:47.195 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a3b0cf23a90f6822965') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:47.738 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:279 495ms Sun Nov 24 14:57:47.739 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:991052 496ms Sun Nov 24 14:57:47.794 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34333 reslen:16656081 223ms Sun Nov 24 14:57:47.808 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1459, clonedBytes: 265962016, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:47.810 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.811 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.812 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.813 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a3b0cf23a90f6822998') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:47.813 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:47.816 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a3b0cf23a90f6822998') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:48.137 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:268 275ms Sun Nov 24 14:57:48.137 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:244 275ms Sun Nov 24 14:57:48.834 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:49.764 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1195796 598ms Sun Nov 24 14:57:49.845 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:49.846 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:49.848 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:49.849 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:49.850 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a3d0cf23a90f6822a7d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:49.851 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a3d0cf23a90f6822a7d') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:49.862 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:50.578 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:264 696ms Sun Nov 24 14:57:50.579 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1393199 697ms Sun Nov 24 14:57:50.887 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:51.913 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:52.547 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:2996314 1498ms Sun Nov 24 14:57:52.939 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:53.716 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:2095681 1048ms Sun Nov 24 14:57:53.804 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:53.805 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:53.806 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:53.807 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:53.807 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a410cf23a90f6822b63') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:53.809 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a410cf23a90f6822b63') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:53.965 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:54.330 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:989572 495ms Sun Nov 24 14:57:54.389 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:54.390 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:54.391 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:54.392 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:54.393 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a420cf23a90f6822b8f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:54.394 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a420cf23a90f6822b8f') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:54.991 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:55.064 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1229719 615ms Sun Nov 24 14:57:55.849 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1330098 665ms Sun Nov 24 14:57:56.019 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:56.651 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:237 683ms Sun Nov 24 14:57:56.651 [conn3625] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:1366710 683ms Sun Nov 24 14:57:56.727 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:56.728 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:56.729 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:56.729 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:57:56.731 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a440cf23a90f6822c76') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:56.731 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a440cf23a90f6822c76') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:57:56.833 [conn3643] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a440cf23a90f6822c76') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:383 101ms Sun Nov 24 14:57:56.833 [conn10] command admin.$cmd command: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a440cf23a90f6822c76') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } ntoreturn:1 keyUpdates:0 reslen:383 101ms Sun Nov 24 14:57:57.045 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:57.164 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:656989 329ms Sun Nov 24 14:57:58.071 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:57:59.097 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:00.122 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:01.148 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:01.302 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:6533967 3267ms Sun Nov 24 14:58:01.729 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:624401 312ms Sun Nov 24 14:58:01.792 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:01.793 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:01.793 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:01.794 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:01.795 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a490cf23a90f6822d5a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:01.797 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a490cf23a90f6822d5a') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:02.173 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:02.405 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:02.406 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:02.409 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a4a0cf23a90f6822d87') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:02.411 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:02.414 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:02.416 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a4a0cf23a90f6822d87') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:03.198 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:03.457 [conn3640] insert dicomdb.dicom ninserted:1 keyUpdates:0 locks(micros) w:223 963ms Sun Nov 24 14:58:03.458 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1927026 964ms Sun Nov 24 14:58:04.223 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:04.577 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:2003482 1002ms Sun Nov 24 14:58:05.248 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:05.450 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1496056 748ms Sun Nov 24 14:58:05.495 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:05.496 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:05.499 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a4d0cf23a90f6822e72') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:05.500 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:05.502 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:05.505 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a4d0cf23a90f6822e72') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:06.273 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:06.342 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1546928 774ms Sun Nov 24 14:58:07.150 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1383165 692ms Sun Nov 24 14:58:07.150 [conn3643] command admin.$cmd command: { splitVector: "dicomdb.dicom", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('529258de0cf23a90f681aff5') }, max: { files_id: MaxKey }, maxChunkSizeBytes: 67108864, maxSplitPoints: 2, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 locks(micros) r:90 reslen:53 701ms Sun Nov 24 14:58:07.298 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:07.880 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1227461 614ms Sun Nov 24 14:58:07.964 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:07.965 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:07.967 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:07.967 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a4f0cf23a90f6822f59') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:07.968 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:07.970 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a4f0cf23a90f6822f59') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:08.323 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:08.730 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:08.731 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:08.732 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:08.732 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:08.733 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a500cf23a90f6822f88') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:08.734 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a500cf23a90f6822f88') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:09.348 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:09.458 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1361255 680ms Sun Nov 24 14:58:10.373 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:10.415 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1679199 840ms Sun Nov 24 14:58:11.358 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1646289 823ms Sun Nov 24 14:58:11.398 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1508, clonedBytes: 276640410, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:11.449 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:11.450 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:11.451 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:11.452 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:11.453 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a530cf23a90f6823071') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:11.455 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a530cf23a90f6823071') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:12.423 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1515, clonedBytes: 278199136, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:12.654 [conn3640] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1047329 524ms Sun Nov 24 14:58:13.441 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:215 668ms Sun Nov 24 14:58:13.442 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:208 668ms Sun Nov 24 14:58:13.484 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1548, clonedBytes: 284787340, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:13.957 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:793987 397ms Sun Nov 24 14:58:13.966 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:2657 reslen:858143 287ms Sun Nov 24 14:58:13.969 [conn10] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:13.970 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:13.970 [conn10] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:13.971 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:13.972 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a550cf23a90f6823157') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:13.973 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a550cf23a90f6823157') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:14.052 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:14.053 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:14.055 [conn3643] request split points lookup for chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:14.056 [conn3643] max number of requested split points reached (2) before the end of chunk dicomdb.fs.chunks { : ObjectId('52925a260cf23a90f682214f') } -->> { : MaxKey } Sun Nov 24 14:58:14.057 [conn10] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a560cf23a90f6823184') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:14.058 [conn3643] received splitChunk request: { splitChunk: "dicomdb.fs.chunks", keyPattern: { files_id: 1.0 }, min: { files_id: ObjectId('52925a260cf23a90f682214f') }, max: { files_id: MaxKey }, from: "dicom1", splitKeys: [ { files_id: ObjectId('52925a560cf23a90f6823184') } ], shardId: "dicomdb.fs.chunks-files_id_ObjectId('52925a260cf23a90f682214f')", configdb: "node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020" } Sun Nov 24 14:58:14.509 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "clone", counts: { cloned: 1614, clonedBytes: 299200360, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:14.546 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:948831 475ms Sun Nov 24 14:58:15.313 [conn3625] insert dicomdb.fs.chunks ninserted:1 keyUpdates:0 locks(micros) w:1297744 649ms Sun Nov 24 14:58:15.316 [conn3675] command admin.$cmd command: { _migrateClone: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:48 reslen:51 403ms Sun Nov 24 14:58:15.534 [conn15] moveChunk data transfer progress: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "steady", counts: { cloned: 1618, clonedBytes: 300058440, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 Sun Nov 24 14:58:15.534 [conn15] moveChunk setting version to: 12|0||529258b4c868398bb45e3755 Sun Nov 24 14:58:15.540 [conn15] moveChunk migrate commit accepted by TO-shard: { active: true, ns: "dicomdb.fs.chunks", from: "dicom1/node40.clus.cci.emory.edu:27001", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shardKeyPattern: { files_id: 1.0 }, state: "done", counts: { cloned: 1618, clonedBytes: 300058440, catchup: 0, steady: 0 }, ok: 1.0 } Sun Nov 24 14:58:15.540 [conn15] moveChunk updating self version to: 12|1||529258b4c868398bb45e3755 through { files_id: ObjectId('529259620cf23a90f681d1ec') } -> { files_id: ObjectId('5292597d0cf23a90f681de3f') } for collection 'dicomdb.fs.chunks' Sun Nov 24 14:58:15.904 [conn3640] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:245 464ms Sun Nov 24 14:58:15.904 [conn3625] insert dicomdb.fs.files ninserted:1 keyUpdates:0 locks(micros) w:250 464ms Sun Nov 24 14:58:15.910 [conn9] command admin.$cmd command: { writebacklisten: ObjectId('528f8628c868398bb45de20e') } ntoreturn:1 keyUpdates:0 reslen:262523 50686ms Sun Nov 24 14:58:15.973 [conn3642] waiting till out of critical section Sun Nov 24 14:58:25.540 [conn15] Socket recv() timeout 192.168.1.159:27020 Sun Nov 24 14:58:25.540 [conn15] SocketException: remote: 192.168.1.159:27020 error: 9001 socket exception [3] server [192.168.1.159:27020] Sun Nov 24 14:58:25.540 [conn15] DBClientCursor::init call() failed Sun Nov 24 14:58:25.977 [conn3642] waiting till out of critical section Sun Nov 24 14:58:29.726 [conn15] scoped connection to node40.clus.cci.emory.edu:27020,node40.clus.cci.emory.edu:27021,node41.clus.cci.emory.edu:27020 not being returned to the pool Sun Nov 24 14:58:29.727 [conn15] warning: 13104 SyncClusterConnection::findOne prepare failed: 10276 DBClientBase::findN: transport error: node40.clus.cci.emory.edu:27020 ns: admin.$cmd query: { fsync: 1 } node40.clus.cci.emory.edu:27020:{} Sun Nov 24 14:58:29.727 [conn15] warning: moveChunk commit outcome ongoing: { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')", lastmod: Timestamp 12000|0, lastmodEpoch: ObjectId('529258b4c868398bb45e3755'), ns: "dicomdb.fs.chunks", min: { files_id: ObjectId('5292592d0cf23a90f681c5e0') }, max: { files_id: ObjectId('529259620cf23a90f681d1ec') }, shard: "dicom2" }, o2: { _id: "dicomdb.fs.chunks-files_id_ObjectId('5292592d0cf23a90f681c5e0')" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')", lastmod: Timestamp 12000|1, lastmodEpoch: ObjectId('529258b4c868398bb45e3755'), ns: "dicomdb.fs.chunks", min: { files_id: ObjectId('529259620cf23a90f681d1ec') }, max: { files_id: ObjectId('5292597d0cf23a90f681de3f') }, shard: "dicom1" }, o2: { _id: "dicomdb.fs.chunks-files_id_ObjectId('529259620cf23a90f681d1ec')" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "dicomdb.fs.chunks" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 11000|3 } } ] } for command :{ $err: "SyncClusterConnection::findOne prepare failed: 10276 DBClientBase::findN: transport error: node40.clus.cci.emory.edu:27020 ns: admin.$cmd query: { fsy...", code: 13104 } Sun Nov 24 14:58:35.980 [conn3642] waiting till out of critical section Sun Nov 24 14:58:38.672 [DataFileSync] flushing mmaps took 48713ms for 11 files Sun Nov 24 14:58:39.729 [conn15] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27020] Sun Nov 24 14:58:39.730 [conn15] SyncClusterConnection connecting to [node40.clus.cci.emory.edu:27021] Sun Nov 24 14:58:39.730 [conn15] SyncClusterConnection connecting to [node41.clus.cci.emory.edu:27020] Sun Nov 24 14:58:39.731 [conn15] ERROR: moveChunk commit failed: version is at11|3||000000000000000000000000 instead of 12|1||529258b4c868398bb45e3755 Sun Nov 24 14:58:39.731 [conn15] ERROR: TERMINATING Sun Nov 24 14:58:39.731 dbexit: Sun Nov 24 14:58:39.731 [conn15] shutdown: going to close listening sockets... Sun Nov 24 14:58:39.732 [conn15] closing listening socket: 9 Sun Nov 24 14:58:39.732 [conn15] closing listening socket: 10 Sun Nov 24 14:58:39.732 [conn15] closing listening socket: 11 Sun Nov 24 14:58:39.732 [conn15] removing socket file: /tmp/mongodb-27001.sock Sun Nov 24 14:58:39.732 [conn15] shutdown: going to flush diaglog... Sun Nov 24 14:58:39.732 [conn15] shutdown: going to close sockets... Sun Nov 24 14:58:39.732 [conn15] shutdown: waiting for fs preallocator... Sun Nov 24 14:58:39.732 [conn15] shutdown: lock for final commit... Sun Nov 24 14:58:39.732 [conn15] shutdown: final commit... Sun Nov 24 14:58:39.733 [initandlisten] now exiting Sun Nov 24 14:58:39.733 dbexit: ; exiting immediately