[SERVER-22249] stats.js - Not starting chunk migration because another migration is already in progress Created: 15/Jan/16  Updated: 18/Nov/16  Resolved: 26/Jan/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 3.2.3, 3.3.1

Type: Bug Priority: Major - P3
Reporter: Spencer Jackson Assignee: Dianna Hohensee (Inactive)
Resolution: Done Votes: 0
Labels: test-only
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: Sharding F (01/29/16)
Participants:
Linked BF Score: 0

 Description   

sharding_auth failed on enterprise-windows-64

stats.js - Logs | History

BF Ticket Generated by spencer.jackson



 Comments   
Comment by Githook User [ 26/Jan/16 ]

Author:

{u'username': u'DiannaHohensee', u'name': u'Dianna Hohensee', u'email': u'dianna.hohensee@10gen.com'}

Message: SERVER-22249 disabling balancer in sharding stats.js test because it can conflict with explicit chunk manipulation depending on the timing of operations

(cherry picked from commit 0f61121be2a8baca70f97f95d2089753d00d9f44)
Branch: v3.2
https://github.com/mongodb/mongo/commit/7bb22cd3d5db2521653ad152e7e887dbb0648018

Comment by Githook User [ 21/Jan/16 ]

Author:

{u'username': u'DiannaHohensee', u'name': u'Dianna Hohensee', u'email': u'dianna.hohensee@10gen.com'}

Message: SERVER-22249 disabling balancer in sharding stats.js test because it can conflict with explicit chunk manipulation depending on the timing of operations
Branch: master
https://github.com/mongodb/mongo/commit/0f61121be2a8baca70f97f95d2089753d00d9f44

Comment by Dianna Hohensee (Inactive) [ 20/Jan/16 ]

Sharding's stats.js test has the balancer enabled, while doing explicit chunk manipulation. This leads to an error when attempting to perform the explicit moveChunk operation: it occasionally throws an error because the balancer either has already moved the chunk or is in the process of moving it. Disabling the balancer.

Comment by Spencer Jackson [ 19/Jan/16 ]

Again: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_windows_64_inmem_sharding_auth_3c287ede73fe151729b5fec4801b367453c9f86d_16_01_15_22_16_59
Reassigning to kaloian.manassiev for greater visibility.

Comment by Spencer Jackson [ 15/Jan/16 ]

Again: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_windows_64_sharding_auth_9e99ca9e22553fe7f18f3147af4b41c1da076c7a_16_01_15_18_26_39

Comment by Spencer Jackson [ 15/Jan/16 ]

Again: https://evergreen.mongodb.com/task/mongodb_mongo_v3.2_enterprise_windows_64_inmem_sharding_auth_d1f62c6fbcc0833e3889fa1baa71e30525461c6b_16_01_15_18_36_42
Elevating to blocker.

Comment by Spencer Jackson [ 15/Jan/16 ]

Again: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_windows_64_sharding_WT_ese_0f85256eefa78b1b74b9e6ae0d1c646853f6f22f_16_01_15_17_08_40

Comment by Spencer Jackson [ 15/Jan/16 ]

Again: https://evergreen.mongodb.com/task/mongodb_mongo_master_windows_64_2k8_debug_sharding_WT_00c8c64f6cd9b0d236d50a8a1cf7152e0367a5cd_16_01_15_14_31_26

Comment by Spencer Jackson [ 15/Jan/16 ]

Again: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_windows_64_sharding_WT_ese_82e4da57e076a8830befd1327edfb9f662504e20_16_01_15_14_14_53

Comment by Spencer Jackson [ 15/Jan/16 ]

Again:
https://evergreen.mongodb.com/task/mongodb_mongo_master_windows_64_2k8_ssl_sharding_WT_00c8c64f6cd9b0d236d50a8a1cf7152e0367a5cd_16_01_15_14_31_26
https://logkeeper.mongodb.org/build/5699138f9041300b27470705/test/569928db9041300b274d696f/

[js_test:stats] 2016-01-15T17:14:33.744+0000 2016-01-15T17:14:33.745+0000 E QUERY    [thread1] Error: command {
[js_test:stats] 2016-01-15T17:14:33.744+0000 	"moveChunk" : "test.foo",
[js_test:stats] 2016-01-15T17:14:33.744+0000 	"find" : {
[js_test:stats] 2016-01-15T17:14:33.744+0000 		"_id" : 3
[js_test:stats] 2016-01-15T17:14:33.744+0000 	},
[js_test:stats] 2016-01-15T17:14:33.744+0000 	"to" : "shard0000",
[js_test:stats] 2016-01-15T17:14:33.746+0000 	"_waitForDelete" : true
[js_test:stats] 2016-01-15T17:14:33.746+0000 } failed: { "ok" : 0, "errmsg" : "that chunk is already on that shard" } :
[js_test:stats] 2016-01-15T17:14:33.746+0000 _getErrorWithCode@src/mongo/shell/utils.js:23:13
[js_test:stats] 2016-01-15T17:14:33.746+0000 ShardingTest/this.adminCommand@src/mongo/shell/shardingtest.js:321:15
[js_test:stats] 2016-01-15T17:14:33.746+0000 @jstests\sharding\stats.js:37:1
[js_test:stats] 2016-01-15T17:14:33.746+0000 @jstests\sharding\stats.js:1:2
[js_test:stats] 2016-01-15T17:14:33.746+0000 
[js_test:stats] 2016-01-15T17:14:33.746+0000 failed to load: jstests\sharding\stats.js

Bumping priority.

Comment by Spencer Jackson [ 15/Jan/16 ]

Again:
https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_windows_64_sharding_WT_ese_00c8c64f6cd9b0d236d50a8a1cf7152e0367a5cd_16_01_15_14_31_26
https://logkeeper.mongodb.org/build/5699184ebe07c46e4df5d3ee/test/569940cc9041300b2750e9b0/

[js_test:stats] 2016-01-15T18:56:46.785+0000 2016-01-15T18:56:46.785+0000 E QUERY    [thread1] Error: command {
[js_test:stats] 2016-01-15T18:56:46.785+0000 	"moveChunk" : "test.foo",
[js_test:stats] 2016-01-15T18:56:46.786+0000 	"find" : {
[js_test:stats] 2016-01-15T18:56:46.786+0000 		"_id" : 3
[js_test:stats] 2016-01-15T18:56:46.786+0000 	},
[js_test:stats] 2016-01-15T18:56:46.786+0000 	"to" : "shard0000",
[js_test:stats] 2016-01-15T18:56:46.786+0000 	"_waitForDelete" : true
[js_test:stats] 2016-01-15T18:56:46.786+0000 } failed: { "ok" : 0, "errmsg" : "that chunk is already on that shard" } :
[js_test:stats] 2016-01-15T18:56:46.786+0000 _getErrorWithCode@src/mongo/shell/utils.js:23:13
[js_test:stats] 2016-01-15T18:56:46.786+0000 ShardingTest/this.adminCommand@src/mongo/shell/shardingtest.js:321:15
[js_test:stats] 2016-01-15T18:56:46.786+0000 @jstests\sharding\stats.js:37:1
[js_test:stats] 2016-01-15T18:56:46.786+0000 @jstests\sharding\stats.js:1:2
[js_test:stats] 2016-01-15T18:56:46.786+0000 
[js_test:stats] 2016-01-15T18:56:46.786+0000 failed to load: jstests\sharding\stats.js

Comment by Spencer Jackson [ 15/Jan/16 ]

Hmmm. Another, very likely related issue:
https://evergreen.mongodb.com/task/mongodb_mongo_master_windows_64_sharding_WT_00c8c64f6cd9b0d236d50a8a1cf7152e0367a5cd_16_01_15_14_31_26
https://logkeeper.mongodb.org/build/56991b42be07c46e4df6ed95/test/56993021be07c46e4dfc91a8/

[js_test:stats] 2016-01-15T17:45:37.479+0000 s20015| 2016-01-15T17:45:37.479+0000 I ASIO     [NetworkInterfaceASIO-0] Successfully connected to WIN-P7JBDNSR43T:20012
[js_test:stats] 2016-01-15T17:45:37.480+0000 2016-01-15T17:45:37.480+0000 E QUERY    [thread1] Error: command {
[js_test:stats] 2016-01-15T17:45:37.480+0000 	"moveChunk" : "test.foo",
[js_test:stats] 2016-01-15T17:45:37.480+0000 	"find" : {
[js_test:stats] 2016-01-15T17:45:37.480+0000 		"_id" : 3
[js_test:stats] 2016-01-15T17:45:37.480+0000 	},
[js_test:stats] 2016-01-15T17:45:37.480+0000 	"to" : "shard0000",
[js_test:stats] 2016-01-15T17:45:37.480+0000 	"_waitForDelete" : true
[js_test:stats] 2016-01-15T17:45:37.480+0000 } failed: { "ok" : 0, "errmsg" : "that chunk is already on that shard" } :
[js_test:stats] 2016-01-15T17:45:37.482+0000 _getErrorWithCode@src/mongo/shell/utils.js:23:13
[js_test:stats] 2016-01-15T17:45:37.482+0000 ShardingTest/this.adminCommand@src/mongo/shell/shardingtest.js:321:15
[js_test:stats] 2016-01-15T17:45:37.482+0000 @jstests\sharding\stats.js:37:1
[js_test:stats] 2016-01-15T17:45:37.482+0000 @jstests\sharding\stats.js:1:2
[js_test:stats] 2016-01-15T17:45:37.482+0000 
[js_test:stats] 2016-01-15T17:45:37.482+0000 failed to load: jstests\sharding\stats.js

Comment by Spencer Jackson [ 15/Jan/16 ]

[js_test:stats] 2016-01-15T18:59:25.487+0000 d20011| 2016-01-15T18:59:25.487+0000 I SHARDING [conn6] about to log metadata event into changelog: { _id: "WIN-P7JBDNSR43T-2016-01-15T18:59:25.487+0000-5699418dbdca3292886ef0dd", server: "WIN-P7JBDNSR43T", clientAddr: "10.171.110.60:65523", time: new Date(1452884365487), what: "moveChunk.from", ns: "test.foo", details: { min: { _id: MinKey }, max: { _id: 5000.0 }, step 1 of 6: 0, to: "shard0000", from: "shard0001", note: "aborted" } }
[js_test:stats] 2016-01-15T18:59:25.490+0000 s20015| 2016-01-15T18:59:25.489+0000 I SHARDING [conn1] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
[js_test:stats] 2016-01-15T18:59:25.490+0000 2016-01-15T18:59:25.491+0000 E QUERY    [thread1] Error: command {
[js_test:stats] 2016-01-15T18:59:25.490+0000 	"moveChunk" : "test.foo",
[js_test:stats] 2016-01-15T18:59:25.490+0000 	"find" : {
[js_test:stats] 2016-01-15T18:59:25.490+0000 		"_id" : 3
[js_test:stats] 2016-01-15T18:59:25.490+0000 	},
[js_test:stats] 2016-01-15T18:59:25.490+0000 	"to" : "shard0000",
[js_test:stats] 2016-01-15T18:59:25.492+0000 	"_waitForDelete" : true
[js_test:stats] 2016-01-15T18:59:25.492+0000 } failed: {
[js_test:stats] 2016-01-15T18:59:25.492+0000 	"cause" : {
[js_test:stats] 2016-01-15T18:59:25.492+0000 		"ok" : 0,
[js_test:stats] 2016-01-15T18:59:25.492+0000 		"errmsg" : "Not starting chunk migration because another migration is already in progress",
[js_test:stats] 2016-01-15T18:59:25.492+0000 		"code" : 117
[js_test:stats] 2016-01-15T18:59:25.492+0000 	},
[js_test:stats] 2016-01-15T18:59:25.492+0000 	"code" : 117,
[js_test:stats] 2016-01-15T18:59:25.492+0000 	"ok" : 0,
[js_test:stats] 2016-01-15T18:59:25.493+0000 	"errmsg" : "move failed"
[js_test:stats] 2016-01-15T18:59:25.493+0000 } :
[js_test:stats] 2016-01-15T18:59:25.493+0000 _getErrorWithCode@src/mongo/shell/utils.js:23:13
[js_test:stats] 2016-01-15T18:59:25.493+0000 ShardingTest/this.adminCommand@src/mongo/shell/shardingtest.js:321:15
[js_test:stats] 2016-01-15T18:59:25.493+0000 @jstests\sharding\stats.js:37:1
[js_test:stats] 2016-01-15T18:59:25.493+0000 @jstests\sharding\stats.js:1:2
[js_test:stats] 2016-01-15T18:59:25.493+0000 
[js_test:stats] 2016-01-15T18:59:25.493+0000 failed to load: jstests\sharding\stats.js

Not really sure about this, looks like a general sharding issue, so sending to the sharding backlog.

Generated at Thu Feb 08 03:59:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.