Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-51733

config server in a docker container fails to start up with authentication

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 4.4.0, 4.4.1, 4.7.0
    • Fix Version/s: 4.9.0, 4.4.3
    • Component/s: None
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.4
    • Steps To Reproduce:
      Hide

      First start up a config server in a docker container:

      docker run \
        --env "MONGO_INITDB_ROOT_USERNAME=root" \
        --env "MONGO_INITDB_ROOT_PASSWORD=D0ntTRyTh1satH0me" \
        mongo mongod --configsvr --bind_ip_all --replSet "configrs" --auth
      

      Once it has started up create the repl set

      docker exec -it f03ea4b2ceaf \
        mongo localhost:27019 \
          --authenticationDatabase "admin" \
          -u "root" \
          -p "D0ntTRyTh1satH0me" \
          --eval "rs.initiate( { _id: \"configrs\", configsvr: true, members: [ { _id : 0, host : \"localhost:27019\" } ] } )"
      

      After a few moments the database will crash producing the following error:

       

      {"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F",  "c":"-",        "id":23081,   "ctx":"OplogApplier-0","msg":"Invariant failure","attr":{"expr":"collection->isEmpty(opCtx)","msg":"de8cd9ae-e981-4b25-9a87-ef34d1a3506c","file":"src/mongo/db/index_builds_coordinator.cpp","line":1478}}
       
       
      {"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F",  "c":"-",        "id":23082,   "ctx":"OplogApplier-0","msg":"\n\n***aborting after invariant() failure\n\n"}
       
       
      {"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"OplogApplier-0","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
      {"t":{"$date":"2020-10-19T18:03:30.521+00:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"OplogApplier-0","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"564C52D789F1","b":"564C500B0000","o":"2CC89F1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},{"a":"564C52D7A029","b":"564C500B0000","o":"2CCA029","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"564C52D77886","b":"564C500B0000","o":"2CC7886","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7FC37C68E8A0","b":"7FC37C67C000","o":"128A0","s":"funlockfile","s+":"50"},{"a":"7FC37C2C9F47","b":"7FC37C28B000","o":"3EF47","s":"gsignal","s+":"C7"},{"a":"7FC37C2CB8B1","b":"7FC37C28B000","o":"408B1","s":"abort","s+":"141"},{"a":"564C50FB865E","b":"564C500B0000","o":"F0865E","s":"_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j","s+":"153"},{"a":"564C51A2FC0E","b":"564C500B0000","o":"197FC0E","s":"_ZN5mongo22IndexBuildsCoordinator30createIndexesOnEmptyCollectionEPNS_16OperationContextENS_4UUIDERKSt6vectorINS_7BSONObjESaIS5_EEb","s+":"30E"},{"a":"564C51664944","b":"564C500B0000","o":"15B4944","s":"_ZZN5mongo10ShardLocal19createIndexOnConfigEPNS_16OperationContextERKNS_15NamespaceStringERKNS_7BSONObjEbENKUlvE0_clEv","s+":"64"},{"a":"564C51665521","b":"564C500B0000","o":"15B5521","s":"_ZN5mongo10ShardLocal19createIndexOnConfigEPNS_16OperationContextERKNS_15NamespaceStringERKNS_7BSONObjEb","s+":"4B1"},{"a":"564C516E5991","b":"564C500B0000","o":"1635991","s":"_ZN5mongo22ShardingCatalogManager18_initConfigIndexesEPNS_16OperationContextE","s+":"DE1"},{"a":"564C516E8B04","b":"564C500B0000","o":"1638B04","s":"_ZN5mongo22ShardingCatalogManager32initializeConfigDatabaseIfNeededEPNS_16OperationContextE","s+":"54"},{"a":"564C51328C2A","b":"564C500B0000","o":"1278C2A","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl34_shardingOnTransitionToPrimaryHookEPNS_16OperationContextE","s+":"24A"},{"a":"564C5132A51B","b":"564C500B0000","o":"127A51B","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl21onTransitionToPrimaryEPNS_16OperationContextE","s+":"23B"},{"a":"564C51369703","b":"564C500B0000","o":"12B9703","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx","s+":"423"},{"a":"564C513FB264","b":"564C500B0000","o":"134B264","s":"_ZN5mongo4repl16OplogApplierImpl4_runEPNS0_11OplogBufferE","s+":"904"},{"a":"564C5144A1F8","b":"564C500B0000","o":"139A1F8","s":"_ZZN5mongo15unique_functionIFvRKNS_8executor12TaskExecutor12CallbackArgsEEE8makeImplIZNS_4repl12OplogApplier7startupEvEUlS5_E_EEDaOT_EN12SpecificImpl4callES5_","s+":"F8"},{"a":"564C52706BE3","b":"564C500B0000","o":"2656BE3","s":"_ZN5mongo8executor22ThreadPoolTaskExecutor11runCallbackESt10shared_ptrINS1_13CallbackStateEE","s+":"113"},{"a":"564C52706FF2","b":"564C500B0000","o":"2656FF2","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_8executor22ThreadPoolTaskExecutor23scheduleIntoPool_inlockEPNSt7__cxx114listISt10shared_ptrINS6_13CallbackStateEESaISB_EEERKSt14_List_iteratorISB_ESI_St11unique_lockINS_12latch_detail5LatchEEEUlT_E1_EEDaOSN_EN12SpecificImpl4callEOS1_","s+":"A2"},{"a":"564C528B1762","b":"564C500B0000","o":"2801762","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"},{"a":"564C528B3F86","b":"564C500B0000","o":"2803F86","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"},{"a":"564C528B4D31","b":"564C500B0000","o":"2804D31","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"},{"a":"564C528B5060","b":"564C500B0000","o":"2805060","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"},{"a":"564C52F249DF","b":"564C500B0000","o":"2E749DF","s":"execute_native_thread_routine","s+":"F"},{"a":"7FC37C6836DB","b":"7FC37C67C000","o":"76DB","s":"start_thread","s+":"DB"},{"a":"7FC37C3ACA3F","b":"7FC37C28B000","o":"121A3F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","compiledModules":[],"uname":{"sysname":"Linux","release":"5.8.13-arch1-1","version":"#1 SMP PREEMPT Thu, 01 Oct 2020 20:40:35 +0000","machine":"x86_64"},"somap":[{"b":"564C500B0000","elfType":3,"buildId":"832AC9AE6CC5692FB33A92460CF800BB1B8C176A"},{"b":"7FC37C67C000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"BC3C06107774266C5F7DB3F1F380A3DA68AF90FA"},{"b":"7FC37C28B000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"D3CF764B2F97AC3EFE366DDD07AD902FB6928FD7"}]}}}}
      

      I could not add the complete log of the incident as that exceeds the allowed length in this text box. Therefore you can find it at the following link if needed:

      http://sprunge.us/dgsHqH

       

       

      Show
      First start up a config server in a docker container: docker run \ --env "MONGO_INITDB_ROOT_USERNAME=root" \ --env "MONGO_INITDB_ROOT_PASSWORD=D0ntTRyTh1satH0me" \ mongo mongod --configsvr --bind_ip_all --replSet "configrs" --auth Once it has started up create the repl set docker exec -it f03ea4b2ceaf \ mongo localhost: 27019 \ --authenticationDatabase "admin" \ -u "root" \ -p "D0ntTRyTh1satH0me" \ --eval "rs.initiate( { _id: \"configrs\", configsvr: true, members: [ { _id : 0, host : \"localhost:27019\" } ] } )" After a few moments the database will crash producing the following error:   {"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F", "c":"-", "id":23081, "ctx":"OplogApplier-0","msg":"Invariant failure","attr":{"expr":"collection->isEmpty(opCtx)","msg":"de8cd9ae-e981-4b25-9a87-ef34d1a3506c","file":"src/mongo/db/index_builds_coordinator.cpp","line":1478}}     {"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F", "c":"-", "id":23082, "ctx":"OplogApplier-0","msg":"\n\n***aborting after invariant() failure\n\n"}     {"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"OplogApplier-0","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} {"t":{"$date":"2020-10-19T18:03:30.521+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"OplogApplier-0","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"564C52D789F1","b":"564C500B0000","o":"2CC89F1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},{"a":"564C52D7A029","b":"564C500B0000","o":"2CCA029","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"564C52D77886","b":"564C500B0000","o":"2CC7886","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7FC37C68E8A0","b":"7FC37C67C000","o":"128A0","s":"funlockfile","s+":"50"},{"a":"7FC37C2C9F47","b":"7FC37C28B000","o":"3EF47","s":"gsignal","s+":"C7"},{"a":"7FC37C2CB8B1","b":"7FC37C28B000","o":"408B1","s":"abort","s+":"141"},{"a":"564C50FB865E","b":"564C500B0000","o":"F0865E","s":"_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j","s+":"153"},{"a":"564C51A2FC0E","b":"564C500B0000","o":"197FC0E","s":"_ZN5mongo22IndexBuildsCoordinator30createIndexesOnEmptyCollectionEPNS_16OperationContextENS_4UUIDERKSt6vectorINS_7BSONObjESaIS5_EEb","s+":"30E"},{"a":"564C51664944","b":"564C500B0000","o":"15B4944","s":"_ZZN5mongo10ShardLocal19createIndexOnConfigEPNS_16OperationContextERKNS_15NamespaceStringERKNS_7BSONObjEbENKUlvE0_clEv","s+":"64"},{"a":"564C51665521","b":"564C500B0000","o":"15B5521","s":"_ZN5mongo10ShardLocal19createIndexOnConfigEPNS_16OperationContextERKNS_15NamespaceStringERKNS_7BSONObjEb","s+":"4B1"},{"a":"564C516E5991","b":"564C500B0000","o":"1635991","s":"_ZN5mongo22ShardingCatalogManager18_initConfigIndexesEPNS_16OperationContextE","s+":"DE1"},{"a":"564C516E8B04","b":"564C500B0000","o":"1638B04","s":"_ZN5mongo22ShardingCatalogManager32initializeConfigDatabaseIfNeededEPNS_16OperationContextE","s+":"54"},{"a":"564C51328C2A","b":"564C500B0000","o":"1278C2A","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl34_shardingOnTransitionToPrimaryHookEPNS_16OperationContextE","s+":"24A"},{"a":"564C5132A51B","b":"564C500B0000","o":"127A51B","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl21onTransitionToPrimaryEPNS_16OperationContextE","s+":"23B"},{"a":"564C51369703","b":"564C500B0000","o":"12B9703","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx","s+":"423"},{"a":"564C513FB264","b":"564C500B0000","o":"134B264","s":"_ZN5mongo4repl16OplogApplierImpl4_runEPNS0_11OplogBufferE","s+":"904"},{"a":"564C5144A1F8","b":"564C500B0000","o":"139A1F8","s":"_ZZN5mongo15unique_functionIFvRKNS_8executor12TaskExecutor12CallbackArgsEEE8makeImplIZNS_4repl12OplogApplier7startupEvEUlS5_E_EEDaOT_EN12SpecificImpl4callES5_","s+":"F8"},{"a":"564C52706BE3","b":"564C500B0000","o":"2656BE3","s":"_ZN5mongo8executor22ThreadPoolTaskExecutor11runCallbackESt10shared_ptrINS1_13CallbackStateEE","s+":"113"},{"a":"564C52706FF2","b":"564C500B0000","o":"2656FF2","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_8executor22ThreadPoolTaskExecutor23scheduleIntoPool_inlockEPNSt7__cxx114listISt10shared_ptrINS6_13CallbackStateEESaISB_EEERKSt14_List_iteratorISB_ESI_St11unique_lockINS_12latch_detail5LatchEEEUlT_E1_EEDaOSN_EN12SpecificImpl4callEOS1_","s+":"A2"},{"a":"564C528B1762","b":"564C500B0000","o":"2801762","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"},{"a":"564C528B3F86","b":"564C500B0000","o":"2803F86","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"},{"a":"564C528B4D31","b":"564C500B0000","o":"2804D31","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"},{"a":"564C528B5060","b":"564C500B0000","o":"2805060","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"},{"a":"564C52F249DF","b":"564C500B0000","o":"2E749DF","s":"execute_native_thread_routine","s+":"F"},{"a":"7FC37C6836DB","b":"7FC37C67C000","o":"76DB","s":"start_thread","s+":"DB"},{"a":"7FC37C3ACA3F","b":"7FC37C28B000","o":"121A3F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","compiledModules":[],"uname":{"sysname":"Linux","release":"5.8.13-arch1-1","version":"#1 SMP PREEMPT Thu, 01 Oct 2020 20:40:35 +0000","machine":"x86_64"},"somap":[{"b":"564C500B0000","elfType":3,"buildId":"832AC9AE6CC5692FB33A92460CF800BB1B8C176A"},{"b":"7FC37C67C000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"BC3C06107774266C5F7DB3F1F380A3DA68AF90FA"},{"b":"7FC37C28B000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"D3CF764B2F97AC3EFE366DDD07AD902FB6928FD7"}]}}}} I could not add the complete log of the incident as that exceeds the allowed length in this text box. Therefore you can find it at the following link if needed: http://sprunge.us/dgsHqH    
    • Sprint:
      Execution Team 2020-11-02

      Description

      When mongo is running inside a docker container it is impossible to create a new replica set on a fresh config server.

      I have tried to find a work around by trying this on versions 4.4.1, 4.7.0, and 4.4.0, all produce the same error. I have also tried running it in a custom built container on Ubuntu 20.04 (focal) and also get the same error.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              benety.goh Benety Goh
              Reporter:
              freemo@gmail.com Jeffrey Phillips Freeman
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: