[SERVER-22347] Mongo shell replica set connections must use legacy readMode Created: 29/Jan/16  Updated: 08/Jan/24  Resolved: 17/Jun/16

Status: Closed
Project: Core Server
Component/s: Querying, Shell
Affects Version/s: 3.2.1
Fix Version/s: 3.2.9, 3.3.9

Type: Bug Priority: Major - P3
Reporter: Peter Assignee: Max Hirschhorn
Resolution: Done Votes: 0
Labels: code-and-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-24135 just mongo shell exit Closed
Related
related to SERVER-23219 DBCommandCursor doesn't route getMore... Closed
related to SERVER-33284 Garbage collection of DBCommandCursor... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

unknown

Sprint: Query 11 (03/14/16), Query 12 (04/04/16), Query 16 (06/24/16)
Participants:

 Description   

Opened shell to create index. It was created successfully but on exit I received trace:

$ mongo --host  atv/mongo1,mongo2,mongo3
MongoDB shell version: 3.2.1
connecting to: atv/mongo1,mongo2,mongo3:27017/test
2016-01-29T07:21:45.890+0300 I NETWORK  [thread1] Starting new replica set monitor for atv/mongo1:27017,mongo2:27017,mongo3:27017
2016-01-29T07:21:45.890+0300 I NETWORK  [ReplicaSetMonitorWatcher] starting
atv:PRIMARY> show dbs
atv-content        0.389GB
[other dbs here]
atv:PRIMARY> use atv-content
switched to db atv-content
atv:PRIMARY> db.content.find()
[documents]
Type "it" for more
atv:PRIMARY> db.programs.createIndex({"updated": -1}, {"background": true})
{
	"createdCollectionAutomatically" : false,
	"numIndexesBefore" : 18,
	"numIndexesAfter" : 19,
	"ok" : 1
}
atv:PRIMARY> 
bye
2016-01-29T07:46:19.152+0300 I -        [thread1] Assertion failure 0 src/mongo/client/dbclient_rs.cpp 610
2016-01-29T07:46:19.159+0300 I CONTROL  [thread1] 
 0x55cfd3593e6c 0x55cfd3523a8c 0x55cfd3513896 0x55cfd333934e 0x55cfd348d38f 0x55cfd3b4527a 0x55cfd3adbb4c 0x55cfd3b45534 0x55cfd3addd00 0x55cfd3af5bab 0x55cfd3af6392 0x55cfd3af7121 0x55cfd3af7b51 0x55cfd3af7dcd 0x55cfd3af8033 0x55cfd3a80eac 0x55cfd3495a53 0x55cfd34961f0 0x55cfd349632f 0x55cfd34c4a6e 0x55cfd3483dc4 0x7f6f81110b93 0x7f6f8090d51c 0x7f6f8064f8bd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55CFD2EE9000","o":"6AAE6C"},{"b":"55CFD2EE9000","o":"63AA8C"},{"b":"55CFD2EE9000","o":"62A896"},{"b":"55CFD2EE9000","o":"45034E"},{"b":"55CFD2EE9000","o":"5A438F"},{"b":"55CFD2EE9000","o":"C5C27A"},{"b":"55CFD2EE9000","o":"BF2B4C"},{"b":"55CFD2EE9000","o":"C5C534"},{"b":"55CFD2EE9000","o":"BF4D00"},{"b":"55CFD2EE9000","o":"C0CBAB"},{"b":"55CFD2EE9000","o":"C0D392"},{"b":"55CFD2EE9000","o":"C0E121"},{"b":"55CFD2EE9000","o":"C0EB51"},{"b":"55CFD2EE9000","o":"C0EDCD"},{"b":"55CFD2EE9000","o":"C0F033"},{"b":"55CFD2EE9000","o":"B97EAC"},{"b":"55CFD2EE9000","o":"5ACA53"},{"b":"55CFD2EE9000","o":"5AD1F0"},{"b":"55CFD2EE9000","o":"5AD32F"},{"b":"55CFD2EE9000","o":"5DBA6E"},{"b":"55CFD2EE9000","o":"59ADC4"},{"b":"7F6F8103E000","o":"D2B93"},{"b":"7F6F80906000","o":"751C"},{"b":"7F6F8055B000","o":"F48BD"}],"processInfo":{ "mongodbVersion" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.1.9-gentoo-r1", "version" : "#2 SMP Mon Oct 19 11:22:39 MSK 2015", "machine" : "x86_64" }, "somap" : [ { "b" : "55CFD2EE9000", "elfType" : 3 }, { "b" : "7FFD5FBDF000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "224DCF70CA2B5365E9243240CECE1D02BEABC9BD" }, { "b" : "7F6F82EA4000", "path" : "/usr/lib64/libboost_program_options.so.1.58.0", "elfType" : 3 }, { "b" : "7F6F82C8A000", "path" : "/usr/lib64/libboost_filesystem.so.1.58.0", "elfType" : 3 }, { "b" : "7F6F82A64000", "path" : "/usr/lib64/libboost_thread.so.1.58.0", "elfType" : 3 }, { "b" : "7F6F82778000", "path" : "/usr/lib64/libboost_regex.so.1.58.0", "elfType" : 3 }, { "b" : "7F6F82560000", "path" : "/lib64/libz.so.1", "elfType" : 3 }, { "b" : "7F6F822D9000", "path" : "/usr/lib64/libyaml-cpp.so.0.5", "elfType" : 3 }, { "b" : "7F6F82082000", "path" : "/usr/lib64/libstemmer.so.0d", "elfType" : 3 }, { "b" : "7F6F81E77000", "path" : "/usr/lib64/libpcrecpp.so.0", "elfType" : 3 }, { "b" : "7F6F81BF7000", "path" : "/usr/lib64/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7F6F81778000", "path" : "/usr/lib64/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7F6F81570000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F6F8136C000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F6F8103E000", "path" : "/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F6F80D3A000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F6F80B23000", "path" : "/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F6F80906000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F6F8055B000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F6F83122000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7F6F80357000", "path" : "/usr/lib64/libboost_system.so.1.58.0", "elfType" : 3 }, { "b" : "7F6F80111000", "path" : "/lib64/libpcre.so.1", "elfType" : 3 } ] }}
 mongo(_ZN5mongo15printStackTraceERSo+0x4C) [0x55cfd3593e6c]
 mongo(_ZN5mongo10logContextEPKc+0x14C) [0x55cfd3523a8c]
 mongo(_ZN5mongo12verifyFailedEPKcS1_j+0xD6) [0x55cfd3513896]
 mongo(+0x45034E) [0x55cfd333934e]
 mongo(_ZN5mongo5mozjs16CursorHandleInfo8finalizeEP8JSFreeOpP8JSObject+0xAF) [0x55cfd348d38f]
 mongo(_ZN2js2gc5Arena8finalizeI8JSObjectEEmPNS_6FreeOpENS0_9AllocKindEm+0x24A) [0x55cfd3b4527a]
 mongo(+0xBF2B4C) [0x55cfd3adbb4c]
 mongo(_ZN2js2gc10ArenaLists16forceFinalizeNowEPNS_6FreeOpENS0_9AllocKindENS1_14KeepArenasEnumEPPNS0_11ArenaHeaderE+0x114) [0x55cfd3b45534]
 mongo(_ZN2js2gc10ArenaLists30queueForegroundObjectsForSweepEPNS_6FreeOpE+0x90) [0x55cfd3addd00]
 mongo(_ZN2js2gc9GCRuntime22beginSweepingZoneGroupEv+0x8FB) [0x55cfd3af5bab]
 mongo(_ZN2js2gc9GCRuntime15beginSweepPhaseEb+0x1D2) [0x55cfd3af6392]
 mongo(_ZN2js2gc9GCRuntime23incrementalCollectSliceERNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x281) [0x55cfd3af7121]
 mongo(_ZN2js2gc9GCRuntime7gcCycleEbRNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x1B1) [0x55cfd3af7b51]
 mongo(_ZN2js2gc9GCRuntime7collectEbNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x12D) [0x55cfd3af7dcd]
 mongo(_ZN2js2gc9GCRuntime2gcE18JSGCInvocationKindN2JS8gcreason6ReasonE+0x53) [0x55cfd3af8033]
 mongo(_ZN2js14DestroyContextEP9JSContextNS_18DestroyContextModeE+0x14C) [0x55cfd3a80eac]
 mongo(_ZN5mongo5mozjs14MozJSImplScope10MozRuntimeD1Ev+0x33) [0x55cfd3495a53]
 mongo(_ZN5mongo5mozjs14MozJSImplScopeD1Ev+0x720) [0x55cfd34961f0]
 mongo(_ZN5mongo5mozjs14MozJSImplScopeD0Ev+0x2F) [0x55cfd349632f]
 mongo(_ZN5mongo5mozjs15MozJSProxyScope10implThreadEPv+0x18E) [0x55cfd34c4a6e]
 mongo(_ZN4nspr6Thread13ThreadRoutineEPv+0x44) [0x55cfd3483dc4]
 libstdc++.so.6(+0xD2B93) [0x7f6f81110b93]
 libpthread.so.0(+0x751C) [0x7f6f8090d51c]
 libc.so.6(clone+0x6D) [0x7f6f8064f8bd]
-----  END BACKTRACE  -----
2016-01-29T07:46:19.159+0300 I QUERY    [thread1] Failed to kill cursor 35276858153 due to UnknownError assertion src/mongo/client/dbclient_rs.cpp:610

I've tried to reproduce this issue (dropped index, created again, ...), but failed. Still I wanted to show you this.



 Comments   
Comment by Githook User [ 29/Jul/16 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-22347 Use "legacy" read mode for replica set connections.

Adds a Mongo.prototype.isReplicaSetConnection() method to connection
objects in the mongo shell. The mongo shell then degrades to use
"legacy" read mode for replica set connections because DBCommandCursor
doesn't route getMore and killCursors operations to the original server.

(cherry picked from commit ef08c9cb7a040c36f367269edb1f144a063f1ee1)
Branch: v3.2
https://github.com/mongodb/mongo/commit/5f288387f694706a14eedde8ab910ce234bc47b9

Comment by Githook User [ 17/Jun/16 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-22347 Use "legacy" read mode for replica set connections.

Adds a Mongo.prototype.isReplicaSetConnection() method to connection
objects in the mongo shell. The mongo shell then degrades to use
"legacy" read mode for replica set connections because DBCommandCursor
doesn't route getMore and killCursors operations to the original server.
Branch: master
https://github.com/mongodb/mongo/commit/ef08c9cb7a040c36f367269edb1f144a063f1ee1

Comment by David Storch [ 10/May/16 ]

Our current plan is to fix this issue by forcing the shell to use legacy readMode when it is using a replica set connection. A more complete fix to the underlying issue is tracked in SERVER-23219.

Comment by Max Hirschhorn [ 18/Mar/16 ]

Per discussion with jason.rassi, we've decided to make a change to DBClientReplicaSet so that it will issue the killCursors operation to the current primary of the replica set. As explained in an earlier comment, this approach is flawed because

  • a cursor established on the current primary will continue to exist following a stepdown
  • a cursor can be established on a node other than the primary if slaveOk is set on the connection

However, it is consistent with our handling for getMore operations under the aforementioned scenarios. I've filed SERVER-23219 to address the fundamental issue.

Generated at Thu Feb 08 04:00:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.