Loading...

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: 3.1.9
Affects Version/s: None
Component/s: Sharding
Labels:
None

Backwards Compatibility:
Fully Compatible
Operating System:
ALL
Sprint:
Sharding A (10/09/15)

An invariant failure in sharding_initialization.cpp can be triggered by running the removeShard command against mongos under certain circumstance. Affects development branch only.

Link to example test failures of authCommands.js from the past 24 hours:

Linux / sharding_WT: task, logs
SSL Ubuntu 14.04: task, logs

Excerpt from first failure above:

[js_test:authCommands] 2015-09-25T03:52:30.520-0400 Running command that should succeed:
[js_test:authCommands] 2015-09-25T03:52:30.520-0400 { "removeshard" : "test-rs1" }
[js_test:authCommands] 2015-09-25T03:52:30.521-0400 d20011| 2015-09-25T03:52:30.520-0400 I COMMAND  [repl writer worker 12] dropDatabase test finished
[js_test:authCommands] 2015-09-25T03:52:30.521-0400 d20012| 2015-09-25T03:52:30.521-0400 I COMMAND  [repl writer worker 10] dropDatabase test finished
[js_test:authCommands] 2015-09-25T03:52:30.573-0400 s20019| 2015-09-25T03:52:30.573-0400 I SHARDING [conn1] going to start draining shard: test-rs1
[js_test:authCommands] 2015-09-25T03:52:30.652-0400 s20019| 2015-09-25T03:52:30.652-0400 I SHARDING [conn1] about to log metadata event: { _id: "build13.ny.cbi.10gen.cc-2015-09-25T03:52:30.652-0400-5604fd3efd7f69035a6d70d9", server: "build13.ny.cbi.10gen.cc", clientAddr: "127.0.0.1:64465", time: new Date(1443167550652), what: "removeShard.start", ns: "", details: { shard: "test-rs1" } }
[js_test:authCommands] 2015-09-25T03:52:30.707-0400 {
[js_test:authCommands] 2015-09-25T03:52:30.707-0400 	"msg" : "draining started successfully",
[js_test:authCommands] 2015-09-25T03:52:30.708-0400 	"state" : "started",
[js_test:authCommands] 2015-09-25T03:52:30.708-0400 	"shard" : "test-rs1",
[js_test:authCommands] 2015-09-25T03:52:30.708-0400 	"note" : "you need to drop or movePrimary these databases",
[js_test:authCommands] 2015-09-25T03:52:30.708-0400 	"dbsToMove" : [ ],
[js_test:authCommands] 2015-09-25T03:52:30.708-0400 	"ok" : 1
[js_test:authCommands] 2015-09-25T03:52:30.708-0400 }
[js_test:authCommands] 2015-09-25T03:52:30.709-0400 Running command that should succeed:
[js_test:authCommands] 2015-09-25T03:52:30.709-0400 { "removeshard" : "test-rs1" }
[js_test:authCommands] 2015-09-25T03:52:30.711-0400 s20019| 2015-09-25T03:52:30.711-0400 I SHARDING [conn1] going to remove shard: test-rs1
[js_test:authCommands] 2015-09-25T03:52:30.713-0400 d20013| 2015-09-25T03:52:30.713-0400 I NETWORK  [conn23] end connection 127.0.0.1:57254 (20 connections now open)
[js_test:authCommands] 2015-09-25T03:52:30.714-0400 d20013| 2015-09-25T03:52:30.713-0400 I NETWORK  [conn22] end connection 127.0.0.1:57253 (19 connections now open)
[js_test:authCommands] 2015-09-25T03:52:30.723-0400 s20019| 2015-09-25T03:52:30.723-0400 I -        [thread2] Invariant failure shard src/mongo/s/sharding_initialization.cpp 83
[js_test:authCommands] 2015-09-25T03:52:30.723-0400 s20019| 2015-09-25T03:52:30.723-0400 I -        [thread2]
[js_test:authCommands] 2015-09-25T03:52:30.723-0400 s20019|
[js_test:authCommands] 2015-09-25T03:52:30.724-0400 s20019| ***aborting after invariant() failure
[js_test:authCommands] 2015-09-25T03:52:30.724-0400 s20019|
[js_test:authCommands] 2015-09-25T03:52:30.724-0400 s20019|
[js_test:authCommands] 2015-09-25T03:52:30.733-0400 s20019| 2015-09-25T03:52:30.733-0400 F -        [thread2] Got signal: 6 (Aborted).
[js_test:authCommands] 2015-09-25T03:52:30.733-0400 s20019|
[js_test:authCommands] 2015-09-25T03:52:30.733-0400 s20019|  0xbaf522 0xbae439 0xbaec42 0x7ff14c515b10 0x7ff14c1e0265 0x7ff14c1e1d10 0xb489cb 0xb00513 0x968bab 0x961494 0x9590b4 0x94b78b 0x949057 0x94a641 0x94a789 0x955fae 0xbcc3e9 0xbcc661 0xbd07ef 0x953d2f 0xd14c80 0x7ff14c50d73d 0x7ff14c283d1d
[js_test:authCommands] 2015-09-25T03:52:30.733-0400 s20019| ----- BEGIN BACKTRACE -----
[js_test:authCommands] 2015-09-25T03:52:30.735-0400 s20019| {"backtrace":[{"b":"400000","o":"7AF522"},{"b":"400000","o":"7AE439"},{"b":"400000","o":"7AEC42"},{"b":"7FF14C507000","o":"EB10"},{"b":"7FF14C1B0000","o":"30265"},{"b":"7FF14C1B0000","o":"31D10"},{"b":"400000","o":"7489CB"},{"b":"400000","o":"700513"},{"b":"400000","o":"568BAB"},{"b":"400000","o":"561494"},{"b":"400000","o":"5590B4"},{"b":"400000","o":"54B78B"},{"b":"400000","o":"549057"},{"b":"400000","o":"54A641"},{"b":"400000","o":"54A789"},{"b":"400000","o":"555FAE"},{"b":"400000","o":"7CC3E9"},{"b":"400000","o":"7CC661"},{"b":"400000","o":"7D07EF"},{"b":"400000","o":"553D2F"},{"b":"400000","o":"914C80"},{"b":"7FF14C507000","o":"673D"},{"b":"7FF14C1B0000","o":"D3D1D"}],"processInfo":{ "mongodbVersion" : "3.1.9-pre-", "gitVersion" : "4898cb582633fd686c3057824ce0d1713284d15d", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-431.3.1.el6.x86_64", "version" : "#1 SMP Fri Jan 3 21:39:27 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFAD436000", "elfType" : 3 }, { "b" : "7FF14D0B7000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FF14CEB3000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FF14CBB3000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FF14C930000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FF14C722000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FF14C507000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FF14C1B0000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FF14D2C0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
[js_test:authCommands] 2015-09-25T03:52:30.735-0400 s20019|  mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xbaf522]
[js_test:authCommands] 2015-09-25T03:52:30.735-0400 s20019|  mongos(+0x7AE439) [0xbae439]
[js_test:authCommands] 2015-09-25T03:52:30.736-0400 s20019|  mongos(+0x7AEC42) [0xbaec42]
[js_test:authCommands] 2015-09-25T03:52:30.736-0400 s20019|  libpthread.so.0(+0xEB10) [0x7ff14c515b10]
[js_test:authCommands] 2015-09-25T03:52:30.736-0400 s20019|  libc.so.6(gsignal+0x35) [0x7ff14c1e0265]
[js_test:authCommands] 2015-09-25T03:52:30.736-0400 s20019|  libc.so.6(abort+0x110) [0x7ff14c1e1d10]
[js_test:authCommands] 2015-09-25T03:52:30.736-0400 s20019|  mongos(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0xb489cb]
[js_test:authCommands] 2015-09-25T03:52:30.737-0400 s20019|  mongos(+0x700513) [0xb00513]
[js_test:authCommands] 2015-09-25T03:52:30.737-0400 s20019|  mongos(_ZN5mongo8executor20NetworkInterfaceASIO7AsyncOp12beginCommandERKNS0_20RemoteCommandRequestEPNS_3rpc18EgressMetadataHookE+0x22B) [0x968bab]
[js_test:authCommands] 2015-09-25T03:52:30.737-0400 s20019|  mongos(_ZN5mongo8executor20NetworkInterfaceASIO19_beginCommunicationEPNS1_7AsyncOpE+0x124) [0x961494]
[js_test:authCommands] 2015-09-25T03:52:30.737-0400 s20019|  mongos(+0x5590B4) [0x9590b4]
[js_test:authCommands] 2015-09-25T03:52:30.738-0400 s20019|  mongos(_ZNKSt8functionIFvN5mongo10StatusWithISt10unique_ptrINS0_8executor14ConnectionPool19ConnectionInterfaceENS4_23ConnectionHandleDeleterEEEEEEclES8_+0x4B) [0x94b78b]
[js_test:authCommands] 2015-09-25T03:52:30.738-0400 s20019|  mongos(_ZN5mongo8executor14ConnectionPool12SpecificPool15fulfillRequestsERSt11unique_lockISt5mutexE+0x157) [0x949057]
[js_test:authCommands] 2015-09-25T03:52:30.738-0400 s20019|  mongos(_ZN5mongo8executor14ConnectionPool12SpecificPool13getConnectionERKNS_11HostAndPortENSt6chrono8durationIlSt5ratioILl1ELl1000EEEESt11unique_lockISt5mutexESt8functionIFvNS_10StatusWithISt10unique_ptrINS1_19ConnectionInterfaceENS1_23ConnectionHandleDeleterEEEEEE+0x261) [0x94a641]
[js_test:authCommands] 2015-09-25T03:52:30.738-0400 s20019|  mongos(_ZN5mongo8executor14ConnectionPool3getERKNS_11HostAndPortENSt6chrono8durationIlSt5ratioILl1ELl1000EEEESt8functionIFvNS_10StatusWithISt10unique_ptrINS1_19ConnectionInterfaceENS1_23ConnectionHandleDeleterEEEEEE+0xE9) [0x94a789]
[js_test:authCommands] 2015-09-25T03:52:30.739-0400 s20019|  mongos(+0x555FAE) [0x955fae]
[js_test:authCommands] 2015-09-25T03:52:30.739-0400 s20019|  mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x319) [0xbcc3e9]
[js_test:authCommands] 2015-09-25T03:52:30.739-0400 s20019|  mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xC1) [0xbcc661]
[js_test:authCommands] 2015-09-25T03:52:30.739-0400 s20019|  mongos(_ZN4asio10io_service3runEv+0x2F) [0xbd07ef]
[js_test:authCommands] 2015-09-25T03:52:30.739-0400 s20019|  mongos(+0x553D2F) [0x953d2f]
[js_test:authCommands] 2015-09-25T03:52:30.740-0400 s20019|  mongos(+0x914C80) [0xd14c80]
[js_test:authCommands] 2015-09-25T03:52:30.740-0400 s20019|  libpthread.so.0(+0x673D) [0x7ff14c50d73d]
[js_test:authCommands] 2015-09-25T03:52:30.740-0400 s20019|  libc.so.6(clone+0x6D) [0x7ff14c283d1d]
[js_test:authCommands] 2015-09-25T03:52:30.740-0400 s20019| -----  END BACKTRACE  -----
[js_test:authCommands] 2015-09-25T03:52:31.125-0400 c20016| 2015-09-25T03:52:31.125-0400 I NETWORK  [conn9] end connection 127.0.0.1:60812 (17 connections now open)
[js_test:authCommands] 2015-09-25T03:52:31.125-0400 c20017| 2015-09-25T03:52:31.124-0400 I NETWORK  [conn7] end connection 127.0.0.1:48471 (10 connections now open)
[js_test:authCommands] 2015-09-25T03:52:31.125-0400 c20018| 2015-09-25T03:52:31.125-0400 I NETWORK  [conn9] end connection 127.0.0.1:37200 (10 connections now open)
[js_test:authCommands] 2015-09-25T03:52:31.126-0400 c20016| 2015-09-25T03:52:31.125-0400 I NETWORK  [conn11] end connection 127.0.0.1:60814 (17 connections now open)
[js_test:authCommands] 2015-09-25T03:52:31.126-0400 c20016| 2015-09-25T03:52:31.125-0400 I NETWORK  [conn10] end connection 127.0.0.1:60813 (17 connections now open)
[js_test:authCommands] 2015-09-25T03:52:31.126-0400 d20010| 2015-09-25T03:52:31.125-0400 I NETWORK  [conn15] end connection 127.0.0.1:55352 (21 connections now open)
[js_test:authCommands] 2015-09-25T03:52:31.126-0400 2015-09-25T03:52:31.125-0400 E QUERY    [thread1] Error: error doing query: failed :
[js_test:authCommands] 2015-09-25T03:52:31.126-0400 DB.prototype.runCommand@src/mongo/shell/db.js:132:1
[js_test:authCommands] 2015-09-25T03:52:31.126-0400 doTest/checkCommandSucceeded@jstests/sharding/authCommands.js:100:17
[js_test:authCommands] 2015-09-25T03:52:31.127-0400 doTest/checkRemoveShard/checkRemoveShard@jstests/sharding/authCommands.js:237:19
[js_test:authCommands] 2015-09-25T03:52:31.127-0400 assert.soon@src/mongo/shell/assert.js:188:17
[js_test:authCommands] 2015-09-25T03:52:31.127-0400 doTest/checkRemoveShard@jstests/sharding/authCommands.js:240:9
[js_test:authCommands] 2015-09-25T03:52:31.127-0400 doTest@jstests/sharding/authCommands.js:286:1
[js_test:authCommands] 2015-09-25T03:52:31.127-0400 @jstests/sharding/authCommands.js:300:1
[js_test:authCommands] 2015-09-25T03:52:31.127-0400 
[js_test:authCommands] 2015-09-25T03:52:31.128-0400 failed to load: jstests/sharding/authCommands.js

is related to

SERVER-20625 mapreduce.shardedfinish command can trigger invariant if the ShardRegistry is out of date

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates