[SERVER-34414] Creating a role with buildsIndexes:false node hits an fassert Created: 11/Apr/18  Updated: 29/Oct/23  Resolved: 03/Jul/18

Status: Closed
Project: Core Server
Component/s: Replication, Security
Affects Version/s: 3.4.14, 3.6.3
Fix Version/s: 3.4.17, 3.6.7, 4.0.1, 4.1.1

Type: Bug Priority: Major - P3
Reporter: Dugast Fabien Assignee: Matthew Russotto
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
Related
related to SERVER-35962 buildindexes_false_with_system_indexe... Closed
is related to SERVER-34840 Initial sync of a hidden node (buildI... Backlog
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6, v3.4
Sprint: Repl 2018-06-04, Repl 2018-06-18, Repl 2018-07-02
Participants:
Linked BF Score: 34

 Description   

Hello,

I created a fresh replicaset with this command:

doc = {
  '_id': 'test-rs1',
  'members': [
      {
          '_id': 0,
          'host': 'test-mongo-re2.test.foo.com:27017',
      },
      {
          '_id': 1,
          'host': 'test-mongo-re1.test.foo.com:27017',
      },
      {
          '_id': 2,
          'host': 'test-mongo-hid.test.foo.com:27017',
          'buildIndexes': false,
          'hidden': true,
          'priority': 0,
      },
      {
          '_id': 3,
          'host': 'test-mongo-arb.test.foo.com:27017',
          'arbiterOnly': true,
      },
  ]
}
init = rs.initiate(doc);

My Replica is correctly initiated.

As you can see I have a Hidden node.

When I try to create a role in the admin database, with command:

db=db.getSiblingDB('admin');db.createRole({role:'user_rw',roles:[{role:'readWrite',db:'test'}],privileges:[]});

The Hidden node crashes with this stacktrace:

Apr 11 12:16:51 test-mongo-hid.test.foo.com mongod.27017[5013]: [repl writer worker 3] createCollection: admin.system.roles with provided UUID: 9f01b015-241f-433b-9add-783227993953
Apr 11 12:16:51 test-mongo-hid.test.foo.com mongod.27017[5013]: [repl writer worker 3] Fatal assertion 40458 IndexAlreadyExists: no indexes per repl at src/mongo/db/system_index.cpp 224
Apr 11 12:16:51 test-mongo-hid.test.foo.com mongod.27017[5013]: [repl writer worker 3] 
                                                                         
                                                                         ***aborting after fassert() failure
Apr 11 12:16:51 test-mongo-hid.test.foo.com mongod.27017[5013]: [repl writer worker 3] Got signal: 6 (Aborted).
                                                                         
                                                                          0x56389f174f11 0x56389f174129 0x56389f17460d 0x7fa595de3890 0x7fa595a5e067 0x7fa595a5f448 0x56389d8ed763 0x56389dc95bf9 0x56389dc742d5 0x56389dc6e253 0x56389dc755af 0x56389ef3b15e 0x56389de68cfc 0x56389de6a3f6 0x56389de26cf0 0x56389de270b2 0x56389de2956e 0x56389dd4c40c 0x56389dd4211d 0x56389dd43ff5 0x56389dd44746 0x56389dd4485c 0x56389dd4a7e8 0x56389dd4bfe3 0x56389dd406ef 0x56389dd4205a 0x56389e96154c 0x56389e961a4c 0x56389e962436 0x56389f283fd0 0x7fa595ddc064 0x7fa595b1162d
                                                                         ----- BEGIN BACKTRACE -----
                                                                         {"backtrace":[{"b":"56389CF63000","o":"2211F11","s":"_ZN5mongo15printStackTraceERSo"},{"b":"56389CF63000","o":"2211129"},{"b":"56389CF63000","o":"221160D"},{"b":"7FA595DD4000","o":"F890"},{"b":"7FA595A29000","o":"35067","s":"gsignal"},{"b":"7FA595A29000","o":"36448","s":"abort"},{"b":"56389CF63000","o":"98A763","s":"_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj"},{"b":"56389CF63000","o":"D32BF9","s":"_ZN5mongo19createSystemIndexesEPNS_16OperationContextEPNS_10CollectionE"},{"b":"56389CF63000","o":"D112D5","s":"_ZN5mongo12DatabaseImpl16createCollectionEPNS_16OperationContextENS_10StringDataERKNS_17CollectionOptionsEbRKNS_7BSONObjE"},{"b":"56389CF63000","o":"D0B253","s":"_ZN5mongo16userCreateNSImplEPNS_16OperationContextEPNS_8DatabaseENS_10StringDataENS_7BSONObjENS_17CollectionOptions9ParseKindEbRKS5_"},{"b":"56389CF63000","o":"D125AF","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseENS0_10StringDataENS0_7BSONObjENS0_17CollectionOptions9ParseKindEbRKS7_EPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_OS6_OS7_OS9_ObSB_"},{"b":"56389CF63000","o":"1FD815E","s":"_ZN5mongo12userCreateNSEPNS_16OperationContextEPNS_8DatabaseENS_10StringDataENS_7BSONObjENS_17CollectionOptions9ParseKindEbRKS5_"},{"b":"56389CF63000","o":"F05CFC"},{"b":"56389CF63000","o":"F073F6","s":"_ZN5mongo27createCollectionForApplyOpsEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11BSONElementERKNS_7BSONObjESF_"},{"b":"56389CF63000","o":"EC3CF0"},{"b":"56389CF63000","o":"EC40B2"},{"b":"56389CF63000","o":"EC656E","s":"_ZN5mongo4repl19applyCommand_inlockEPNS_16OperationContextERKNS_7BSONObjENS0_16OplogApplication4ModeE"},{"b":"56389CF63000","o":"DE940C","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextERKNS0_7BSONObjENS0_4repl16OplogApplication4ModeEEPSA_E9_M_invokeERKSt9_Any_dataOS3_S6_OS9_"},{"b":"56389CF63000","o":"DDF11D"},{"b":"56389CF63000","o":"DE0FF5","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjENS0_16OplogApplication4ModeESt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS8_S9_IFvvEEEES9_IFSA_S3_S6_S8_EESE_"},{"b":"56389CF63000","o":"DE1746","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjENS0_16OplogApplication4ModeE"},{"b":"56389CF63000","o":"DE185C"},{"b":"56389CF63000","o":"DE77E8","s":"_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjENS0_16OplogApplication4ModeEEE"},{"b":"56389CF63000","o":"DE8FE3","s":"_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE"},{"b":"56389CF63000","o":"DDD6EF"},{"b":"56389CF63000","o":"DDF05A"},{"b":"56389CF63000","o":"19FE54C","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"56389CF63000","o":"19FEA4C","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"56389CF63000","o":"19FF436","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"56389CF63000","o":"2320FD0"},{"b":"7FA595DD4000","o":"8064"},{"b":"7FA595A29000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-5-amd64", "version" : "#1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08)", "machine" : "x86_64" }, "somap" : [ { "b" : "56389CF63000", "elfType" : 3, "buildId" : "F9BA3BD99A1212BD7BCFA9EDA7EB7CC05FEBB67C" }, { "b" : "7FFFF359B000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "8043FE86419956A575F2CFB87B4F76452D8B2911" }, { "b" : "7FA596FE3000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7FA596D79000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FC58685AD5EA6EA755EFD801C425F4CA0D55106B" }, { "b" : "7FA596914000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FA4F70D7CF4870160E8AE5CF7622A2CAB107C848" }, { "b" : "7FA59670C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7FA596508000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7FA596207000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7FA595FF1000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7FA595DD4000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7FA595A29000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7FA5971FA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
                                                                          mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56389f174f11]
                                                                          mongod(+0x2211129) [0x56389f174129]
                                                                          mongod(+0x221160D) [0x56389f17460d]
                                                                          libpthread.so.0(+0xF890) [0x7fa595de3890]
                                                                          libc.so.6(gsignal+0x37) [0x7fa595a5e067]
                                                                          libc.so.6(abort+0x148) [0x7fa595a5f448]
                                                                          mongod(_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj+0x0) [0x56389d8ed763]
                                                                          mongod(_ZN5mongo19createSystemIndexesEPNS_16OperationContextEPNS_10CollectionE+0x3D9) [0x56389dc95bf9]
                                                                          mongod(_ZN5mongo12DatabaseImpl16createCollectionEPNS_16OperationContextENS_10StringDataERKNS_17CollectionOptionsEbRKNS_7BSONObjE+0x845) [0x56389dc742d5]
                                                                          mongod(_ZN5mongo16userCreateNSImplEPNS_16OperationContextEPNS_8DatabaseENS_10StringDataENS_7BSONObjENS_17CollectionOptions9ParseKindEbRKS5_+0xC33) [0x56389dc6e253]
                                                                          mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseENS0_10StringDataENS0_7BSONObjENS0_17CollectionOptions9ParseKindEbRKS7_EPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_OS6_OS7_OS9_ObSB_+0x6F) [0x56389dc755af]
                                                                          mongod(_ZN5mongo12userCreateNSEPNS_16OperationContextEPNS_8DatabaseENS_10StringDataENS_7BSONObjENS_17CollectionOptions9ParseKindEbRKS5_+0x8E) [0x56389ef3b15e]
                                                                          mongod(+0xF05CFC) [0x56389de68cfc]
                                                                          mongod(_ZN5mongo27createCollectionForApplyOpsEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11BSONElementERKNS_7BSONObjESF_+0x236) [0x56389de6a3f6]
                                                                          mongod(+0xEC3CF0) [0x56389de26cf0]
                                                                          mongod(+0xEC40B2) [0x56389de270b2]
                                                                          mongod(_ZN5mongo4repl19applyCommand_inlockEPNS_16OperationContextERKNS_7BSONObjENS0_16OplogApplication4ModeE+0x7AE) [0x56389de2956e]
                                                                          mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextERKNS0_7BSONObjENS0_4repl16OplogApplication4ModeEEPSA_E9_M_invokeERKSt9_Any_dataOS3_S6_OS9_+0x2C) [0x56389dd4c40c]
                                                                          mongod(+0xDDF11D) [0x56389dd4211d]
                                                                          mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjENS0_16OplogApplication4ModeESt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS8_S9_IFvvEEEES9_IFSA_S3_S6_S8_EESE_+0x215) [0x56389dd43ff5]
                                                                          mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjENS0_16OplogApplication4ModeE+0x106) [0x56389dd44746]
                                                                          mongod(+0xDE185C) [0x56389dd4485c]
                                                                          mongod(_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjENS0_16OplogApplication4ModeEEE+0x1C8) [0x56389dd4a7e8]
                                                                          mongod(_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE+0x73) [0x56389dd4bfe3]
                                                                          mongod(+0xDDD6EF) [0x56389dd406ef]
                                                                          mongod(+0xDDF05A) [0x56389dd4205a]
                                                                          mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x14C) [0x56389e96154c]
                                                                          mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xBC) [0x56389e961a4c]
                                                                          mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x96) [0x56389e962436]
                                                                          mongod(+0x2320FD0) [0x56389f283fd0]
                                                                          libpthread.so.0(+0x8064) [0x7fa595ddc064]
                                                                          libc.so.6(clone+0x6D) [0x7fa595b1162d]
                                                                         -----  END BACKTRACE  -----
Apr 11 12:16:51 test-mongo-hid.test.foo.com systemd[1]: mongod@test-rs1.service: Main process exited, code=killed, status=6/ABRT
Apr 11 12:16:51 test-mongo-hid.test.foo.com systemd[1]: mongod@test-rs1.service: Unit entered failed state.
Apr 11 12:16:51 test-mongo-hid.test.foo.com systemd[1]: mongod@test-rs1.service: Failed with result 'signal'.

I can't figure out if it's me who have misconfigured something or if its an internal Mongodb bug.
Any Idea ?



 Comments   
Comment by Githook User [ 03/Aug/18 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@10gen.com', 'username': 'mtrussotto'}

Message: SERVER-34414 Create system indexes using the normal index creation and replication process.
Do not create them directly on secondaries.
Do create oplog entries for index creation.
Branch: v3.4
https://github.com/mongodb/mongo/commit/110d340afde6dd834bf38299d5889cd9c8f38867

Comment by Githook User [ 23/Jul/18 ]

Author:

{'username': 'mtrussotto', 'name': 'Matthew Russotto', 'email': 'matthew.russotto@10gen.com'}

Message: SERVER-34414 Create system indexes using the normal index creation and replication process.
Do not create them directly on secondaries.
Do create oplog entries for index creation.

(cherry picked from commit cfa75c933e125b41596c80b9a1297bb12d318ac3)
Branch: v3.6
https://github.com/mongodb/mongo/commit/deb1161a6c3feb0497c9af4c7b72b01c6d4bdb97

Comment by Githook User [ 19/Jul/18 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@10gen.com', 'username': 'mtrussotto'}

Message: SERVER-34414 Create system indexes using the normal index creation and replication process.
Do not create them directly on secondaries.
Do create oplog entries for index creation.

(cherry picked from commit 7c8d941c7f6904a65476ca91c6013067d2149fe8)
Branch: v4.0
https://github.com/mongodb/mongo/commit/cfa75c933e125b41596c80b9a1297bb12d318ac3

Comment by Githook User [ 03/Jul/18 ]

Author:

{'username': 'mtrussotto', 'name': 'Matthew Russotto', 'email': 'matthew.russotto@10gen.com'}

Message: SERVER-34414 Create system indexes using the normal index creation and replication process.
Do not create them directly on secondaries.
Do create oplog entries for index creation.
Branch: master
https://github.com/mongodb/mongo/commit/7c8d941c7f6904a65476ca91c6013067d2149fe8

Comment by Dugast Fabien [ 12/Apr/18 ]

FYI, The issue occur also when creating user on a fresh replica set. User is created from $external database which store data in admin.user. So I understand that the problem in on the admin database ?

We manage to bypass the issue by creating a dummy role and dummy user before adding a hidden into the RS.

Good luck and thanks !!

Comment by Eric Milkie [ 11/Apr/18 ]

The problem seems to stem from the way system_indexes.cpp is currently written, which is to build system indexes for user auth "in the moment" (when their absence is discovered). and not generate oplog entries for such builds. I'm not sure if we should switch them to generate oplog entries, or if we should simply add another exception for them in the "buildsIndexes" validation logic.

Comment by Eric Milkie [ 11/Apr/18 ]

The crash is actually occurring because you have made the hidden node "buildsIndexes: false", and that is interacting badly with the user authentication feature. It's definitely a bug. Thanks for reporting it! We'll investigate.

Comment by Dugast Fabien [ 11/Apr/18 ]

For more information,
When I create a replica set without the hidden node then I add a the hidden node, then I create the role, I still have the stacktrace.
When I create a replica set without the hidden node then I add a role, then I add the hidden node, then I add a new role, there is no stacktrace and everything is good

Generated at Thu Feb 08 04:36:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.