Tue Aug 3 03:38:42 [conn4] autosplitting mytest.repl size: 189228588 shard: ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 36|4 min: { _id: ObjectId('4c578f2c3a4f3f1b2b00aa6f') } max: { _id: MaxKey } on: { _id: ObjectId('4c578f423a4f3f1b2b00b1d1') }(splitThreshold 188743680) Tue Aug 3 03:38:42 [conn4] config change: { _id: "vm-config1.saas-sd.lava.tw-2010-8-3_3-38-35", server: "vm-config1.saas-sd.lava.tw", time: new Date(1280806722086), what: "split", ns: "mytest.repl", details: { before: { min: { _id: ObjectId('4c578f2c3a4f3f1b2b00aa6f') }, max: { _id: MaxKey } }, left: { min: { _id: ObjectId('4c578f2c3a4f3f1b2b00aa6f') }, max: { _id: ObjectId('4c578f423a4f3f1b2b00b1d1') } }, right: { min: { _id: ObjectId('4c578f423a4f3f1b2b00b1d1') }, max: { _id: MaxKey } } } } Tue Aug 3 03:39:01 [conn4] autosplitting mytest.repl size: 189228588 shard: ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 37|5 min: { _id: ObjectId('4c578f423a4f3f1b2b00b1d1') } max: { _id: MaxKey } on: { _id: ObjectId('4c578f553a4f3f1b2b00b933') }(splitThreshold 188743680) Tue Aug 3 03:39:01 [conn4] config change: { _id: "vm-config1.saas-sd.lava.tw-2010-8-3_3-39-36", server: "vm-config1.saas-sd.lava.tw", time: new Date(1280806741147), what: "split", ns: "mytest.repl", details: { before: { min: { _id: ObjectId('4c578f423a4f3f1b2b00b1d1') }, max: { _id: MaxKey } }, left: { min: { _id: ObjectId('4c578f423a4f3f1b2b00b1d1') }, max: { _id: ObjectId('4c578f553a4f3f1b2b00b933') } }, right: { min: { _id: ObjectId('4c578f553a4f3f1b2b00b933') }, max: { _id: MaxKey } } } } Tue Aug 3 03:39:01 [Balancer] chose [shard0000] to [shard0001] { _id: "mytest.repl-_id_ObjectId('4c578d083a4f3f1b2b000000')", lastmod: Timestamp 4000|0, ns: "mytest.repl", min: { _id: ObjectId('4c578d083a4f3f1b2b000000') }, max: { _id: ObjectId('4c578d0f3a4f3f1b2b000268') }, shard: "shard0000" } Tue Aug 3 03:39:01 [Balancer] moving chunk ns: mytest.repl moving ( ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 4|0 min: { _id: ObjectId('4c578d083a4f3f1b2b000000') } max: { _id: ObjectId('4c578d0f3a4f3f1b2b000268') }) shard0000:shard1/vm-shard11:27018,vm-shard12:27018 -> shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 Tue Aug 3 03:39:15 [conn4] ~ScopedDBConnection: _conn != null Tue Aug 3 03:39:15 [conn4] ERROR: splitIfShould failed: ns: mytest.repl findOne has stale config Tue Aug 3 03:39:16 [Balancer] chose [shard0000] to [shard0001] { _id: "mytest.repl-_id_ObjectId('4c578d0f3a4f3f1b2b000268')", lastmod: Timestamp 4000|4, max: { _id: ObjectId('4c578d143a4f3f1b2b000443') }, min: { _id: ObjectId('4c578d0f3a4f3f1b2b000268') }, ns: "mytest.repl", shard: "shard0000" } Tue Aug 3 03:39:16 [Balancer] moving chunk ns: mytest.repl moving ( ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 4|4 min: { _id: ObjectId('4c578d0f3a4f3f1b2b000268') } max: { _id: ObjectId('4c578d143a4f3f1b2b000443') }) shard0000:shard1/vm-shard11:27018,vm-shard12:27018 -> shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 Tue Aug 3 03:39:28 [Balancer] chose [shard0000] to [shard0001] { _id: "mytest.repl-_id_ObjectId('4c578d1e3a4f3f1b2b0007f9')", lastmod: Timestamp 8000|0, ns: "mytest.repl", min: { _id: ObjectId('4c578d1e3a4f3f1b2b0007f9') }, max: { _id: ObjectId('4c578d213a4f3f1b2b0009d4') }, shard: "shard0000" } Tue Aug 3 03:39:28 [Balancer] moving chunk ns: mytest.repl moving ( ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 8|0 min: { _id: ObjectId('4c578d1e3a4f3f1b2b0007f9') } max: { _id: ObjectId('4c578d213a4f3f1b2b0009d4') }) shard0000:shard1/vm-shard11:27018,vm-shard12:27018 -> shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 Tue Aug 3 03:39:31 [conn4] autosplitting mytest.repl size: 209747972 shard: ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 38|10 min: { _id: ObjectId('4c578f553a4f3f1b2b00b933') } max: { _id: MaxKey } on: { _id: ObjectId('4c578f733a4f3f1b2b00c1b2') }(splitThreshold 188743680) Tue Aug 3 03:39:31 [conn4] dist_lock forcefully taking over from: { _id: "mytest.repl", process: "vm-shard11.saas-sd.lava.tw:1280788106:716187966", state: 1, ts: ObjectId('4c5748ef0e5824b153e4817f'), when: new Date(1280788719639), who: "vm-shard11.saas-sd.lava.tw:1280788106:716187966:conn14:70817093", why: "migrate-{ _id: ObjectId('4c578d1e3a4f3f1b2b0007f9') }" } elapsed minutes: Thu Jan 1 00:00:00 1970 Tue Aug 3 03:39:31 [conn4] config change: { _id: "vm-config1.saas-sd.lava.tw-2010-8-3_3-39-37", server: "vm-config1.saas-sd.lava.tw", time: new Date(1280806771922), what: "split", ns: "mytest.repl", details: { before: { min: { _id: ObjectId('4c578f553a4f3f1b2b00b933') }, max: { _id: MaxKey } }, left: { min: { _id: ObjectId('4c578f553a4f3f1b2b00b933') }, max: { _id: ObjectId('4c578f733a4f3f1b2b00c1b2') } }, right: { min: { _id: ObjectId('4c578f733a4f3f1b2b00c1b2') }, max: { _id: MaxKey } } } } Tue Aug 3 03:39:34 [Balancer] MOVE FAILED **** db assertion failure { assertion: "assertion s/d_migrate.cpp:602", errmsg: "db assertion failure", ok: 0.0 } from: shard0000 to: shard0001 chunk: { _id: "mytest.repl-_id_ObjectId('4c578d1e3a4f3f1b2b0007f9')", lastmod: Timestamp 8000|0, ns: "mytest.repl", min: { _id: ObjectId('4c578d1e3a4f3f1b2b0007f9') }, max: { _id: ObjectId('4c578d213a4f3f1b2b0009d4') }, shard: "shard0000" } Tue Aug 3 03:39:52 [conn4] autosplitting mytest.repl size: 189228588 shard: ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 39|1 min: { _id: ObjectId('4c578f733a4f3f1b2b00c1b2') } max: { _id: MaxKey } on: { _id: ObjectId('4c578f883a4f3f1b2b00c914') }(splitThreshold 188743680) Tue Aug 3 03:39:52 [conn4] config change: { _id: "vm-config1.saas-sd.lava.tw-2010-8-3_3-39-38", server: "vm-config1.saas-sd.lava.tw", time: new Date(1280806792253), what: "split", ns: "mytest.repl", details: { before: { min: { _id: ObjectId('4c578f733a4f3f1b2b00c1b2') }, max: { _id: MaxKey } }, left: { min: { _id: ObjectId('4c578f733a4f3f1b2b00c1b2') }, max: { _id: ObjectId('4c578f883a4f3f1b2b00c914') } }, right: { min: { _id: ObjectId('4c578f883a4f3f1b2b00c914') }, max: { _id: MaxKey } } } } Tue Aug 3 03:40:08 [conn4] autosplitting mytest.repl size: 189228588 shard: ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 40|2 min: { _id: ObjectId('4c578f883a4f3f1b2b00c914') } max: { _id: MaxKey } on: { _id: ObjectId('4c578f983a4f3f1b2b00d076') }(splitThreshold 188743680) Tue Aug 3 03:40:08 [conn4] config change: { _id: "vm-config1.saas-sd.lava.tw-2010-8-3_3-40-39", server: "vm-config1.saas-sd.lava.tw", time: new Date(1280806808453), what: "split", ns: "mytest.repl", details: { before: { min: { _id: ObjectId('4c578f883a4f3f1b2b00c914') }, max: { _id: MaxKey } }, left: { min: { _id: ObjectId('4c578f883a4f3f1b2b00c914') }, max: { _id: ObjectId('4c578f983a4f3f1b2b00d076') } }, right: { min: { _id: ObjectId('4c578f983a4f3f1b2b00d076') }, max: { _id: MaxKey } } } } Tue Aug 3 03:40:25 [conn4] autosplitting mytest.repl size: 189228588 shard: ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 41|3 min: { _id: ObjectId('4c578f983a4f3f1b2b00d076') } max: { _id: MaxKey } on: { _id: ObjectId('4c578fa93a4f3f1b2b00d7d8') }(splitThreshold 188743680) Tue Aug 3 03:40:25 [conn4] config change: { _id: "vm-config1.saas-sd.lava.tw-2010-8-3_3-40-40", server: "vm-config1.saas-sd.lava.tw", time: new Date(1280806825353), what: "split", ns: "mytest.repl", details: { before: { min: { _id: ObjectId('4c578f983a4f3f1b2b00d076') }, max: { _id: MaxKey } }, left: { min: { _id: ObjectId('4c578f983a4f3f1b2b00d076') }, max: { _id: ObjectId('4c578fa93a4f3f1b2b00d7d8') } }, right: { min: { _id: ObjectId('4c578fa93a4f3f1b2b00d7d8') }, max: { _id: MaxKey } } } } Tue Aug 3 03:40:35 [Balancer] chose [shard0000] to [shard0001] { _id: "mytest.repl-_id_ObjectId('4c578d1e3a4f3f1b2b0007f9')", lastmod: Timestamp 8000|0, ns: "mytest.repl", min: { _id: ObjectId('4c578d1e3a4f3f1b2b0007f9') }, max: { _id: ObjectId('4c578d213a4f3f1b2b0009d4') }, shard: "shard0000" } Tue Aug 3 03:40:35 [Balancer] moving chunk ns: mytest.repl moving ( ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 8|0 min: { _id: ObjectId('4c578d1e3a4f3f1b2b0007f9') } max: { _id: ObjectId('4c578d213a4f3f1b2b0009d4') }) shard0000:shard1/vm-shard11:27018,vm-shard12:27018 -> shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 Tue Aug 3 03:40:57 [conn4] ~ScopedDBConnection: _conn != null Tue Aug 3 03:40:57 [conn4] ERROR: splitIfShould failed: ns: mytest.repl findOne has stale config Tue Aug 3 03:40:59 [Balancer] chose [shard0000] to [shard0001] { _id: "mytest.repl-_id_ObjectId('4c578d213a4f3f1b2b0009d4')", lastmod: Timestamp 8000|4, max: { _id: ObjectId('4c578d263a4f3f1b2b000baf') }, min: { _id: ObjectId('4c578d213a4f3f1b2b0009d4') }, ns: "mytest.repl", shard: "shard0000" } Tue Aug 3 03:40:59 [Balancer] moving chunk ns: mytest.repl moving ( ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 8|4 min: { _id: ObjectId('4c578d213a4f3f1b2b0009d4') } max: { _id: ObjectId('4c578d263a4f3f1b2b000baf') }) shard0000:shard1/vm-shard11:27018,vm-shard12:27018 -> shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 Tue Aug 3 03:41:10 [Balancer] chose [shard0000] to [shard0001] { _id: "mytest.repl-_id_ObjectId('4c578d343a4f3f1b2b000f65')", lastmod: Timestamp 12000|0, ns: "mytest.repl", min: { _id: ObjectId('4c578d343a4f3f1b2b000f65') }, max: { _id: ObjectId('4c578d3a3a4f3f1b2b001316') }, shard: "shard0000" } Tue Aug 3 03:41:10 [Balancer] moving chunk ns: mytest.repl moving ( ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 12|0 min: { _id: ObjectId('4c578d343a4f3f1b2b000f65') } max: { _id: ObjectId('4c578d3a3a4f3f1b2b001316') }) shard0000:shard1/vm-shard11:27018,vm-shard12:27018 -> shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 Tue Aug 3 03:41:23 [conn4] autosplitting mytest.repl size: 209754100 shard: ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 42|8 min: { _id: ObjectId('4c578fa93a4f3f1b2b00d7d8') } max: { _id: MaxKey } on: { _id: ObjectId('4c578fe33a4f3f1b2b00e058') }(splitThreshold 188743680) Tue Aug 3 03:41:23 [conn4] dist_lock forcefully taking over from: { _id: "mytest.repl", process: "vm-shard11.saas-sd.lava.tw:1280788106:716187966", state: 1, ts: ObjectId('4c5749550e5824b153e48241'), when: new Date(1280788821844), who: "vm-shard11.saas-sd.lava.tw:1280788106:716187966:conn15:207822227", why: "migrate-{ _id: ObjectId('4c578d343a4f3f1b2b000f65') }" } elapsed minutes: Thu Jan 1 00:00:00 1970 Tue Aug 3 03:41:23 [conn4] config change: { _id: "vm-config1.saas-sd.lava.tw-2010-8-3_3-41-41", server: "vm-config1.saas-sd.lava.tw", time: new Date(1280806883151), what: "split", ns: "mytest.repl", details: { before: { min: { _id: ObjectId('4c578fa93a4f3f1b2b00d7d8') }, max: { _id: MaxKey } }, left: { min: { _id: ObjectId('4c578fa93a4f3f1b2b00d7d8') }, max: { _id: ObjectId('4c578fe33a4f3f1b2b00e058') } }, right: { min: { _id: ObjectId('4c578fe33a4f3f1b2b00e058') }, max: { _id: MaxKey } } } } Tue Aug 3 03:41:23 [conn4] moving chunk (auto): ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 43|1 min: { _id: ObjectId('4c578fe33a4f3f1b2b00e058') } max: { _id: MaxKey } to: shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 #objects: 1 Tue Aug 3 03:41:23 [conn4] moving chunk ns: mytest.repl moving ( ns:mytest.repl at: shard0000:shard1/vm-shard11:27018,vm-shard12:27018 lastmod: 43|1 min: { _id: ObjectId('4c578fe33a4f3f1b2b00e058') } max: { _id: MaxKey }) shard0000:shard1/vm-shard11:27018,vm-shard12:27018 -> shard0001:shard2/vm-shard21:27018,vm-shard22:27018,vm-arbiter2:27018 Tue Aug 3 03:41:23 [conn4] Assertion: 10412:moveAndCommit failed: db assertion failure { assertion: "assertion s/d_migrate.cpp:239", errmsg: "db assertion failure", ok: 0.0 } Tue Aug 3 03:41:23 [conn4] ERROR: splitIfShould failed: moveAndCommit failed: db assertion failure { assertion: "assertion s/d_migrate.cpp:239", errmsg: "db assertion failure", ok: 0.0 } Tue Aug 3 03:41:23 [conn4] end connection 172.16.1.191:45185 Tue Aug 3 03:41:24 [Balancer] MOVE FAILED **** db assertion failure { assertion: "assertion s/d_migrate.cpp:602", errmsg: "db assertion failure", ok: 0.0 } from: shard0000 to: shard0001 chunk: { _id: "mytest.repl-_id_ObjectId('4c578d343a4f3f1b2b000f65')", lastmod: Timestamp 12000|0, ns: "mytest.repl", min: { _id: ObjectId('4c578d343a4f3f1b2b000f65') }, max: { _id: ObjectId('4c578d3a3a4f3f1b2b001316') }, shard: "shard0000" } Tue Aug 3 03:44:28 [conn5] end connection 172.16.1.193:47075