[SERVER-21119] Better behavior for shell with readMode "commands" and find() when switching to an older server Created: 26/Oct/15  Updated: 01/Feb/16  Resolved: 01/Feb/16

Status: Closed
Project: Core Server
Component/s: Querying, Shell
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Joanna Cheng Assignee: Jonathan Reams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-20915 Resolve shell readMode on connection ... Closed
Backwards Compatibility: Fully Compatible
Sprint: Build C (11/20/15), Build F (01/29/16)
Participants:

 Description   

If someone is downgrading, they could easily hit this scenario.

In addition to the error in the shell, there's also a scary looking stack trace in the logs when you run this for the first time

2015-10-26T12:48:36.827+1100 I QUERY    [conn9]     operation isn't supported: 2008
2015-10-26T12:48:36.827+1100 I -        [conn9] Assertion: 16141:cannot translate opcode 2008
2015-10-26T12:48:36.831+1100 I CONTROL  [conn9]
 0x1050f9449 0x1050ab600 0x1050997da 0x10509969a 0x104aad47a 0x104aa9fd4 0x104c43980 0x104984e3e 0x1050bb39c 0x10512c4f1 0x7fff8e06005a 0x7fff8e05ffd7 0x7fff8e05d3ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"104979000","o":"780449"},{"b":"104979000","o":"732600"},{"b":"104979000","o":"7207DA"},{"b":"104979000","o":"72069A"},{"b":"104979000","o":"13447A"},{"b":"104979000","o":"130FD4"},{"b":"104979000","o":"2CA980"},{"b":"104979000","o":"BE3E"},{"b":"104979000","o":"74239C"},{"b":"104979000","o":"7B34F1"},{"b":"7FFF8E05C000","o":"405A"},{"b":"7FFF8E05C000","o":"3FD7"},{"b":"7FFF8E05C000","o":"13ED"}],"processInfo":{ "mongodbVersion" : "3.0.7", "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd", "uname" : { "sysname" : "Darwin", "release" : "14.5.0", "version" : "Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/usr/local/bin/mongod", "machType" : 2, "b" : "104979000", "buildId" : "C7961951A8C135B2B56385D289485317" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8D24D000", "buildId" : "1866C519C5F33D098C17A8F703664521" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF9236E000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF8FBCA000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF93E49000", "buildId" : "E789748DF9A73CFFB31790DF348B1E95" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF89B6B000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF897EA000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF93F4F000", "buildId" : "5779FFA04D9A3AD4B7F2618227621DC8" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8E284000", "buildId" : "502CF32B669B3709886208188225E4F0" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8FC96000", "buildId" : "CFBBE540D5033AFCB5D6644F1E69949B" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8EFF1000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF8E1BA000", "buildId" : "4F81CA3AD2CE3030A89D42F3DAD7BA8F" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8C788000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF93E79000", "buildId" : "7AF900412768378A925AD83161863642" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF95A34000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF95A1D000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8D090000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8AA40000", "buildId" : "F0635E0FFE4B34DBACF9A58C1E9070E9" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF941AC000", "buildId" : "56F94DCEDBDE36158F07DE6270D9F8BE" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF8FBFB000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF90DAB000", "buildId" : "155DA0A92046332EBFA3D7974A51F731" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF94038000", "buildId" : "0CEB5910843F315CA1DE5D955A48A045" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF898D0000", "buildId" : "2E16C4B3A32739579C41143911979A1E" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF8CE55000", "buildId" : "16AD15EF3DAE3F639D2626CCE1920762" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF94041000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF8E9F3000", "buildId" : "DDA8928BCC0D3255BD8A3FEA0982B890" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8FE22000", "buildId" : "6105C13467223C0AA4CE5E1261E2E1CC" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF95D2D000", "buildId" : "BA58B30B83773B0A8AE34F84021D9D4E" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8BD99000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF91308000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8E05C000", "buildId" : "ACE90967ECD03251AEEB461E3C6414F7" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF96AA9000", "buildId" : "3F5E973FC70231AC97BC05F5C195683C" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF923E0000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF94032000", "buildId" : "D0E968373CF6323DB7116DF6F660E530" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF91690000", "buildId" : "840F5301B55A307890B9FEFFD6CD741A" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF93224000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF8CEB4000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF94D63000", "buildId" : "5C829202962E37448B5000D38CC88E84" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF936C8000", "buildId" : "759E155DBC423D4E869B6F57D477177C" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF90DBD000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF8D00B000", "buildId" : "88A22A0F87C63002BFBAAC0F2808B8B9" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8CB58000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x1050f9449]
 mongod(_ZN5mongo10logContextEPKc+0x100) [0x1050ab600]
 mongod(_ZN5mongo11msgassertedEiPKc+0x13A) [0x1050997da]
 mongod(_ZN5mongo11msgassertedEiRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE+0x1A) [0x10509969a]
 mongod(_ZN5mongo10opToStringEi+0x18A) [0x104aad47a]
 mongod(_ZNK5mongo7OpDebug6reportERKNS_5CurOpERKNS_9LockStatsIxEE+0xB4) [0x104aa9fd4]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xD20) [0x104c43980]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xDE) [0x104984e3e]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x33C) [0x1050bb39c]
 mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x10512c4f1]
 libsystem_pthread.dylib(_pthread_body+0x83) [0x7fff8e06005a]
 libsystem_pthread.dylib(_pthread_body+0x0) [0x7fff8e05ffd7]
 libsystem_pthread.dylib(thread_start+0xD) [0x7fff8e05d3ed]
-----  END BACKTRACE  -----
2015-10-26T12:48:36.831+1100 I NETWORK  [conn9] AssertionException handling request, closing client connection: 16141 cannot translate opcode 2008

Subsequent find() commands give the same message in the shell, but don't generate another stack trace

Workarounds

  1. Restart the shell. I think even restarting with a 3.2.0-rc0 shell works, as the shell will determine it needs be be in "legacy" mode
  2. Manually set the readMode

    db.getMongo().forceReadMode("legacy")
    

    or

    db.getMongo().forceReadMode("compatibility")
    

It would be nice if the shell could figure this out for itself.



 Comments   
Comment by Jonathan Reams [ 01/Feb/16 ]

Since this relates to compatibility between RC's that have already gone GA, and because it doesn't actually cause a failure, just a warning message in the logs - I'm closing this.

Comment by Jonathan Abrahams [ 16/Nov/15 ]

mongo RC-2/mongod RC-3 compatibility issue uncovered belongs in SERVER-20884, and the details have been documented there.

Comment by David Storch [ 16/Nov/15 ]

Got it. We made a performance enhancement under SERVER-20884 which breaks compatibility between 3.2.0-rc3 and previous 3.2.0 release candidates. You shouldn't be able to reproduce this problem unless the shell and server versions span 289d3fdf5c9d5.

Comment by Jonathan Abrahams [ 16/Nov/15 ]

Steps performed on OSX:

  1. wget https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-latest.tgz
  2. tar -xzf mongodb-osx-x86_64-latest.tgz
  3. mongodb-osx-x86_64-3.2.0-rc2-206-g7790034/bin/mongod --dbpath /tmp --port 34567
  4. wget https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-3.2.0-rc2.tgz
  5. tar -xzf mongodb-osx-x86_64-3.2.0-rc2.tgz
  6. mongodb-osx-x86_64-3.2.0-rc2/bin/mongo --port 34567
Comment by David Storch [ 16/Nov/15 ]

jonathan.abrahams, can you provide repro details for the behavior you report? I just did the following

  • Issue a query from a 3.2.0-rc2 shell to a 3.2.0-rc2 server.
  • Stop the server, without closing the shell, and start a 3.0.7 server.
  • Re-issue the query.

This fails for me, as before, with the "cannot translate opcode" error. This happens because the shell is attempting to send a find command using the OP_COMMAND protocol, which is not recognized by a 3.0.7 server. I still think that the resolution of this ticket will be to provide better behavior under SERVER-20915.

Comment by Jonathan Abrahams [ 16/Nov/15 ]

I am unable to connect to mongod (master, v3.2.0-rc2-189-g442fdaa) from mongo shell (3.2.0-rc2) with this error:

mongo 52.91.136.249
MongoDB shell version: 3.2.0-rc2
connecting to: 52.91.136.249/test
2015-11-16T09:52:25.571-0500 E QUERY    [thread1] Error: error doing query: failed :
DB.prototype.runCommand@src/mongo/shell/db.js:132:1
DB.prototype.adminCommand@src/mongo/shell/db.js:149:12
shellHelper.show@src/mongo/shell/utils.js:772:23
shellHelper@src/mongo/shell/utils.js:591:15
@(shellwarnings:1:1
 
2015-11-16T09:52:25.573-0500 I NETWORK  [thread1] trying reconnect to 52.91.136.249:27017 (52.91.136.249) failed
2015-11-16T09:52:25.602-0500 I NETWORK  [thread1] reconnect 52.91.136.249:27017 (52.91.136.249) ok
> show dbs
2015-11-16T09:52:40.946-0500 I NETWORK  [thread1] trying reconnect to 52.91.136.249:27017 (52.91.136.249) failed
2015-11-16T09:52:41.004-0500 I NETWORK  [thread1] reconnect 52.91.136.249:27017 (52.91.136.249) ok
2015-11-16T09:52:41.017-0500 E QUERY    [thread1] Error: error doing query: failed :
DB.prototype.runCommand@src/mongo/shell/db.js:132:1
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:51:15
shellHelper.show@src/mongo/shell/utils.js:697:19
shellHelper@src/mongo/shell/utils.js:591:15
@(shellhelp2):1:1
 
2015-11-16T09:52:41.020-0500 I NETWORK  [thread1] trying reconnect to 52.91.136.249:27017 (52.91.136.249) failed
2015-11-16T09:52:41.045-0500 I NETWORK  [thread1] reconnect 52.91.136.249:27017 (52.91.136.249) ok

I do not think this is an improvement ticket, but now a bug.

Comment by David Storch [ 26/Oct/15 ]

Right now, if a particular readMode is not forced, the shell auto-detects readMode based on the wire version. The result of resolving the readMode is cached for the lifetime of the shell process. This is why the shell currently keeps using readMode "commands" even after making a new connection to an older server that doesn't support the find/getMore commands.

Under SERVER-20915 we are planning to resolve the readMode on connection open. I believe this will cause us to fallback to "legacy" mode when reconnecting to an older server. Assigning to jonathan.reams to confirm that this is the case (and if so, resolve as a duplicate).

Generated at Thu Feb 08 03:56:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.