[SERVER-26440] Invariant failure nss.isValid() when running explain on invalid namespace Created: 03/Oct/16  Updated: 19/Nov/16  Resolved: 01/Nov/16

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 3.3.4
Fix Version/s: 3.4.0-rc3

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Marko Vojvodic
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-23119 Inline map/reduce run through mongos ... Closed
Tested
tested by SERVER-26858 Remove explain_cmd_invalid_namespace.... Closed
Backwards Compatibility: Minor Change
Operating System: ALL
Sprint: Query 2016-10-31, Query 2016-11-21
Participants:
Linked BF Score: 0

 Description   

[ShardedClusterFixture:job0:mongos] 2016-10-03T05:10:22.597+0000 D COMMAND  [conn166] RunOnAllShardsCommand db: test cmd:{ createIndexes: "objid1", indexes: [ { key: { x: 1.0 }, name: "x_1", invalidateUserCache: 1.0 } ] }
[ShardedClusterFixture:job0:mongos] 2016-10-03T05:10:22.615+0000 D SHARDING [conn166] setting shard version of 0|0||000000000000000000000000 for test. on shard shard0001:ip-10-181-30-150:20004
[ShardedClusterFixture:job0:mongos] 2016-10-03T05:10:22.615+0000 I -        [conn166] Invariant failure nss.isValid() src/mongo/s/set_shard_version_request.cpp 102
[ShardedClusterFixture:job0:mongos] 2016-10-03T05:10:22.615+0000 I -        [conn166]
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos] ***aborting after invariant() failure
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos] 2016-10-03T05:10:22.621+0000 F -        [conn166] Got signal: 6 (Aborted).
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos]  0x7f8b2bcb0211 0x7f8b2bcaf309 0x7f8b2bcaf7ed 0x7f8b294937e0 0x7f8b291225e5 0x7f8b29123dc5 0x7f8b2b469e52 0x7f8b2bbbb8b0 0x7f8b2bae3ae2 0x7f8b2bae5f53 0x7f8b2bae7ebf 0x7f8b2baca62d 0x7f8b2b5630f8 0x7f8b2b564b72 0x7f8b2b5696ea 0x7f8b2bb7434f 0x7f8b2bb5c6f7 0x7f8b2bb1072e 0x7f8b2bbb8108 0x7f8b2bb712b4 0x7f8b2bb728a0 0x7f8b2bb6bbd9 0x7f8b2b4b0d69 0x7f8b2bc20fda 0x7f8b2948baa1 0x7f8b291d8aad
[ShardedClusterFixture:job0:mongos] ----- BEGIN BACKTRACE -----
[ShardedClusterFixture:job0:mongos] {"backtrace":[{"b":"7F8B2B078000","o":"C38211","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F8B2B078000","o":"C37309"},{"b":"7F8B2B078000","o":"C377ED"},{"b":"7F8B29484000","o":"F7E0"},{"b":"7F8B290F0000","o":"325E5","s":"gsignal"},{"b":"7F8B290F0000","o":"33DC5","s":"abort"},{"b":"7F8B2B078000","o":"3F1E52","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F8B2B078000","o":"B438B0","s":"_ZN5mongo22SetShardVersionRequest17makeForVersioningERKNS_16ConnectionStringERKNS_7ShardIdES3_RKNS_15NamespaceStringERKNS_12ChunkVersionEb"},{"b":"7F8B2B078000","o":"A6BAE2"},{"b":"7F8B2B078000","o":"A6DF53"},{"b":"7F8B2B078000","o":"A6FEBF","s":"_ZN5mongo14VersionManager19checkShardVersionCBEPNS_16OperationContextEPNS_15ShardConnectionEbi"},{"b":"7F8B2B078000","o":"A5262D","s":"_ZN5mongo15ShardConnection11_finishInitEv"},{"b":"7F8B2B078000","o":"4EB0F8","s":"_ZN5mongo27ParallelSortClusteredCursor28setupVersionAndHandleSlaveOkEPNS_16OperationContextESt10shared_ptrINS_23ParallelConnectionStateEERKNS_7ShardIdES3_INS_5ShardEERKNS_15NamespaceStringERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_INS_12ChunkManagerEE"},{"b":"7F8B2B078000","o":"4ECB72","s":"_ZN5mongo27ParallelSortClusteredCursor9startInitEPNS_16OperationContextE"},{"b":"7F8B2B078000","o":"4F16EA","s":"_ZN5mongo27ParallelSortClusteredCursor4initEPNS_16OperationContextE"},{"b":"7F8B2B078000","o":"AFC34F","s":"_ZN5mongo8Strategy9commandOpEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjEiSA_SD_SD_PSt6vectorINS0_13CommandResultESaISF_EE"},{"b":"7F8B2B078000","o":"AE46F7"},{"b":"7F8B2B078000","o":"A9872E"},{"b":"7F8B2B078000","o":"B40108","s":"_ZN5mongo7Command17execCommandClientEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"7F8B2B078000","o":"AF92B4"},{"b":"7F8B2B078000","o":"AFA8A0","s":"_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE"},{"b":"7F8B2B078000","o":"AF3BD9","s":"_ZN5mongo7Request7processEPNS_16OperationContextEi"},{"b":"7F8B2B078000","o":"438D69","s":"_ZN5mongo23ServiceEntryPointMongos12_sessionLoopEPNS_9transport7SessionE"},{"b":"7F8B2B078000","o":"BA8FDA"},{"b":"7F8B29484000","o":"7AA1"},{"b":"7F8B290F0000","o":"E8AAD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.3.15-44-g445abe2", "gitVersion" : "445abe2e01db50dad295ba8af4971eb243b02546", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-220.el6.x86_64", "version" : "#1 SMP Wed Nov 9 08:03:13 EST 2011", "machine" : "x86_64" }, "somap" : [ { "b" : "7F8B2B078000", "elfType" : 3, "buildId" : "E926DE186841DA672D48D57BD47ADB44ED89023C" }, { "b" : "7FFFE08FF000", "elfType" : 3, "buildId" : "08F634A1D22DEFF00461D50A7699DACDC97657BF" }, { "b" : "7F8B2AC3B000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "E0AEE889D5BF1373F2F9EE0D448DBF3F5B5113F0" }, { "b" : "7F8B2A9F7000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "b" : "7F8B2A7A6000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "1FA3BC4E18EEEB915FDD4E9BE33D0542C3FB2804" }, { "b" : "7F8B2A597000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "244D2593BDE4FE657BC88572DB5DA88FA274B7F3" }, { "b" : "7F8B2A32B000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "b" : "7F8B29F47000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "2CF03CE94B9388A10544E4EF073450851A4D6AEB" }, { "b" : "7F8B29D3F000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "86FE5BC1F46B8F8AA9A7A479FF991900DB93F720" }, { "b" : "7F8B29B3B000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "045D39F19533291EBD72D0EE0247F9D49BE2521E" }, { "b" : "7F8B298B7000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "B60EF3FCE5D4D2D8BAD2585D5CAAA1167B35DBFD" }, { "b" : "7F8B296A1000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "b" : "7F8B29484000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B3BD4C596D72FCBE4607C86FEEC14F47B46D0DCC" }, { "b" : "7F8B290F0000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "24D3AB3DB0F38C7515FEADF82191651DA4117A18" }, { "b" : "7F8B2AE55000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F3EEBD18E66EB139EA4D76CDFA86D643ABCF0070" }, { "b" : "7F8B28ED6000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "DDF6449707FD4C49DDE32A293EEE9AC218BFC460" }, { "b" : "7F8B28C9F000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "B21E32412356755F1851BAE219A0C8EFDAEEC686" }, { "b" : "7F8B289B8000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "b" : "7F8B2878C000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "b" : "7F8B28588000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "13FFCD68952B7715DDF34C9321D82E3041EA9006" }, { "b" : "7F8B2837D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "b" : "7F8B2817A000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "b" : "7F8B27F37000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "9080D18543F337F6F6B5C5265B1A3D2073A0FFBF" }, { "b" : "7F8B27D0A000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "DE75A3731E7ABC427888BA8D38E96606264FBEBB" }, { "b" : "7F8B279CA000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "0375F2A6DA6EDCF870C52584B71798AC9003CFF2" }, { "b" : "7F8B2779E000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "F3A25CFCCA8191255ECFCFCD62248E393AFF3D01" }, { "b" : "7F8B2759A000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "1D3CD12F36DFB9E232953D3B73C34F8C0EF1004D" }, { "b" : "7F8B27395000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "535FB904872A936ECC2E926C612B1B2BFD0FB722" }, { "b" : "7F8B27157000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "29B15E2260EA9A50E0993DEEF7ABD8334F37E6B9" }, { "b" : "7F8B26F41000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7F8B26D3E000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" }, { "b" : "7F8B26B1F000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, { "b" : "7F8B26911000", "path" : "/lib64/libnss_files.so.2", "elfType" : 3, "buildId" : "EED08D874A233A06CDAC0B542C54F991457E1353" }, { "b" : "7F8B2670B000", "path" : "/lib64/libnss_dns.so.2", "elfType" : 3, "buildId" : "925BF6C02DF5D7F588FDF8D47E95B67CF0EC6BC3" } ] }}
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x7f8b2bcb0211]
[ShardedClusterFixture:job0:mongos]  mongos(+0xC37309) [0x7f8b2bcaf309]
[ShardedClusterFixture:job0:mongos]  mongos(+0xC377ED) [0x7f8b2bcaf7ed]
[ShardedClusterFixture:job0:mongos]  libpthread.so.0(+0xF7E0) [0x7f8b294937e0]
[ShardedClusterFixture:job0:mongos]  libc.so.6(gsignal+0x35) [0x7f8b291225e5]
[ShardedClusterFixture:job0:mongos]  libc.so.6(abort+0x175) [0x7f8b29123dc5]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7f8b2b469e52]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo22SetShardVersionRequest17makeForVersioningERKNS_16ConnectionStringERKNS_7ShardIdES3_RKNS_15NamespaceStringERKNS_12ChunkVersionEb+0xD0) [0x7f8b2bbbb8b0]
[ShardedClusterFixture:job0:mongos]  mongos(+0xA6BAE2) [0x7f8b2bae3ae2]
[ShardedClusterFixture:job0:mongos]  mongos(+0xA6DF53) [0x7f8b2bae5f53]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo14VersionManager19checkShardVersionCBEPNS_16OperationContextEPNS_15ShardConnectionEbi+0xBF) [0x7f8b2bae7ebf]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo15ShardConnection11_finishInitEv+0x7D) [0x7f8b2baca62d]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo27ParallelSortClusteredCursor28setupVersionAndHandleSlaveOkEPNS_16OperationContextESt10shared_ptrINS_23ParallelConnectionStateEERKNS_7ShardIdES3_INS_5ShardEERKNS_15NamespaceStringERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_INS_12ChunkManagerEE+0x118) [0x7f8b2b5630f8]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo27ParallelSortClusteredCursor9startInitEPNS_16OperationContextE+0xF02) [0x7f8b2b564b72]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo27ParallelSortClusteredCursor4initEPNS_16OperationContextE+0x2A) [0x7f8b2b5696ea]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo8Strategy9commandOpEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjEiSA_SD_SD_PSt6vectorINS0_13CommandResultESaISF_EE+0x25F) [0x7f8b2bb7434f]
[ShardedClusterFixture:job0:mongos]  mongos(+0xAE46F7) [0x7f8b2bb5c6f7]
[ShardedClusterFixture:job0:mongos]  mongos(+0xA9872E) [0x7f8b2bb1072e]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo7Command17execCommandClientEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0xBD8) [0x7f8b2bbb8108]
[ShardedClusterFixture:job0:mongos]  mongos(+0xAF92B4) [0x7f8b2bb712b4]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE+0xA00) [0x7f8b2bb728a0]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo7Request7processEPNS_16OperationContextEi+0x679) [0x7f8b2bb6bbd9]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo23ServiceEntryPointMongos12_sessionLoopEPNS_9transport7SessionE+0x1F9) [0x7f8b2b4b0d69]
[ShardedClusterFixture:job0:mongos]  mongos(+0xBA8FDA) [0x7f8b2bc20fda]
[ShardedClusterFixture:job0:mongos]  libpthread.so.0(+0x7AA1) [0x7f8b2948baa1]
[ShardedClusterFixture:job0:mongos]  libc.so.6(clone+0x6D) [0x7f8b291d8aad]
[ShardedClusterFixture:job0:mongos] -----  END BACKTRACE  -----
[ShardedClusterFixture:job0:mongos] mongos on port 20005 was expected to be running in teardown(), but wasn't.



 Comments   
Comment by Githook User [ 01/Nov/16 ]

Author:

{u'username': u'm-vojvodic', u'name': u'Marko Vojvodic', u'email': u'marko.vojvodic@mongodb.com'}

Message: SERVER-26440 Validate namespace in DistinctCmd::explain
Branch: master
https://github.com/mongodb/mongo/commit/72b952ddcd3a04a5c8050de435803aac4680a81a

Comment by Kaloian Manassiev [ 07/Oct/16 ]

Without this invariant it is possible to have commands, which expect the namespace to be at a particular version to send invalid namespace and not even realize it, so I am pretty certain that it should be there.

Comment by Max Hirschhorn [ 07/Oct/16 ]

ian.whalen, it is relatively straightforward for the fuzzer to work around triggering this invariant failure; however, I think we should fix the underlying issue before we release 3.4. It seems bad to have a simple user-initiated operation able to crash mongos. We should also see how confident kaloian.manassiev still is in the invariant he added (not that it is wrong, but) because it relies on our command parsing being stringent enough, which from SERVER-23450, SERVER-23480, and SERVER-23889 is known not to always be the case.

Comment by Ian Whalen (Inactive) [ 07/Oct/16 ]

Max do you need this to get fixed for the fuzzer?

Comment by James Wahlin [ 03/Oct/16 ]

Looking again the invariant is not in the createIndexes command. It is unclear whether this is related to SERVER-24032. If a relationship is found the above still stands.

Comment by James Wahlin [ 03/Oct/16 ]

This is likely a side-affect of SERVER-24032. We are now validating createIndexes command content and "invalidateUserCache" is not currently recognized as a valid field. If "invalidateUserCache" as an index specification option serves a purpose other than informational for sharding please see me to discuss.

Generated at Thu Feb 08 04:12:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.