[SERVER-51733] config server in a docker container fails to start up with authentication Created: 19/Oct/20  Updated: 29/Oct/23  Resolved: 27/Oct/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.0, 4.4.1, 4.7.0
Fix Version/s: 4.9.0, 4.4.3

Type: Bug Priority: Major - P3
Reporter: Jeffrey Phillips Freeman Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: Bug
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Duplicate
duplicates SERVER-53259 crash while starting with config serv... Closed
Related
related to SERVER-53259 crash while starting with config serv... Closed
is related to SERVER-18104 Prevent mongos from starting when poi... Closed
is related to SERVER-44437 remove DBDirectClient reference from ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

First start up a config server in a docker container:

docker run \
  --env "MONGO_INITDB_ROOT_USERNAME=root" \
  --env "MONGO_INITDB_ROOT_PASSWORD=D0ntTRyTh1satH0me" \
  mongo mongod --configsvr --bind_ip_all --replSet "configrs" --auth

Once it has started up create the repl set

docker exec -it f03ea4b2ceaf \
  mongo localhost:27019 \
    --authenticationDatabase "admin" \
    -u "root" \
    -p "D0ntTRyTh1satH0me" \
    --eval "rs.initiate( { _id: \"configrs\", configsvr: true, members: [ { _id : 0, host : \"localhost:27019\" } ] } )"

After a few moments the database will crash producing the following error:

 

{"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F",  "c":"-",        "id":23081,   "ctx":"OplogApplier-0","msg":"Invariant failure","attr":{"expr":"collection->isEmpty(opCtx)","msg":"de8cd9ae-e981-4b25-9a87-ef34d1a3506c","file":"src/mongo/db/index_builds_coordinator.cpp","line":1478}}
 
 
{"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F",  "c":"-",        "id":23082,   "ctx":"OplogApplier-0","msg":"\n\n***aborting after invariant() failure\n\n"}
 
 
{"t":{"$date":"2020-10-19T18:03:30.415+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"OplogApplier-0","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2020-10-19T18:03:30.521+00:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"OplogApplier-0","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"564C52D789F1","b":"564C500B0000","o":"2CC89F1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},{"a":"564C52D7A029","b":"564C500B0000","o":"2CCA029","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"564C52D77886","b":"564C500B0000","o":"2CC7886","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7FC37C68E8A0","b":"7FC37C67C000","o":"128A0","s":"funlockfile","s+":"50"},{"a":"7FC37C2C9F47","b":"7FC37C28B000","o":"3EF47","s":"gsignal","s+":"C7"},{"a":"7FC37C2CB8B1","b":"7FC37C28B000","o":"408B1","s":"abort","s+":"141"},{"a":"564C50FB865E","b":"564C500B0000","o":"F0865E","s":"_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j","s+":"153"},{"a":"564C51A2FC0E","b":"564C500B0000","o":"197FC0E","s":"_ZN5mongo22IndexBuildsCoordinator30createIndexesOnEmptyCollectionEPNS_16OperationContextENS_4UUIDERKSt6vectorINS_7BSONObjESaIS5_EEb","s+":"30E"},{"a":"564C51664944","b":"564C500B0000","o":"15B4944","s":"_ZZN5mongo10ShardLocal19createIndexOnConfigEPNS_16OperationContextERKNS_15NamespaceStringERKNS_7BSONObjEbENKUlvE0_clEv","s+":"64"},{"a":"564C51665521","b":"564C500B0000","o":"15B5521","s":"_ZN5mongo10ShardLocal19createIndexOnConfigEPNS_16OperationContextERKNS_15NamespaceStringERKNS_7BSONObjEb","s+":"4B1"},{"a":"564C516E5991","b":"564C500B0000","o":"1635991","s":"_ZN5mongo22ShardingCatalogManager18_initConfigIndexesEPNS_16OperationContextE","s+":"DE1"},{"a":"564C516E8B04","b":"564C500B0000","o":"1638B04","s":"_ZN5mongo22ShardingCatalogManager32initializeConfigDatabaseIfNeededEPNS_16OperationContextE","s+":"54"},{"a":"564C51328C2A","b":"564C500B0000","o":"1278C2A","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl34_shardingOnTransitionToPrimaryHookEPNS_16OperationContextE","s+":"24A"},{"a":"564C5132A51B","b":"564C500B0000","o":"127A51B","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl21onTransitionToPrimaryEPNS_16OperationContextE","s+":"23B"},{"a":"564C51369703","b":"564C500B0000","o":"12B9703","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx","s+":"423"},{"a":"564C513FB264","b":"564C500B0000","o":"134B264","s":"_ZN5mongo4repl16OplogApplierImpl4_runEPNS0_11OplogBufferE","s+":"904"},{"a":"564C5144A1F8","b":"564C500B0000","o":"139A1F8","s":"_ZZN5mongo15unique_functionIFvRKNS_8executor12TaskExecutor12CallbackArgsEEE8makeImplIZNS_4repl12OplogApplier7startupEvEUlS5_E_EEDaOT_EN12SpecificImpl4callES5_","s+":"F8"},{"a":"564C52706BE3","b":"564C500B0000","o":"2656BE3","s":"_ZN5mongo8executor22ThreadPoolTaskExecutor11runCallbackESt10shared_ptrINS1_13CallbackStateEE","s+":"113"},{"a":"564C52706FF2","b":"564C500B0000","o":"2656FF2","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_8executor22ThreadPoolTaskExecutor23scheduleIntoPool_inlockEPNSt7__cxx114listISt10shared_ptrINS6_13CallbackStateEESaISB_EEERKSt14_List_iteratorISB_ESI_St11unique_lockINS_12latch_detail5LatchEEEUlT_E1_EEDaOSN_EN12SpecificImpl4callEOS1_","s+":"A2"},{"a":"564C528B1762","b":"564C500B0000","o":"2801762","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"},{"a":"564C528B3F86","b":"564C500B0000","o":"2803F86","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"},{"a":"564C528B4D31","b":"564C500B0000","o":"2804D31","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"},{"a":"564C528B5060","b":"564C500B0000","o":"2805060","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"},{"a":"564C52F249DF","b":"564C500B0000","o":"2E749DF","s":"execute_native_thread_routine","s+":"F"},{"a":"7FC37C6836DB","b":"7FC37C67C000","o":"76DB","s":"start_thread","s+":"DB"},{"a":"7FC37C3ACA3F","b":"7FC37C28B000","o":"121A3F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","compiledModules":[],"uname":{"sysname":"Linux","release":"5.8.13-arch1-1","version":"#1 SMP PREEMPT Thu, 01 Oct 2020 20:40:35 +0000","machine":"x86_64"},"somap":[{"b":"564C500B0000","elfType":3,"buildId":"832AC9AE6CC5692FB33A92460CF800BB1B8C176A"},{"b":"7FC37C67C000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"BC3C06107774266C5F7DB3F1F380A3DA68AF90FA"},{"b":"7FC37C28B000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"D3CF764B2F97AC3EFE366DDD07AD902FB6928FD7"}]}}}}

I could not add the complete log of the incident as that exceeds the allowed length in this text box. Therefore you can find it at the following link if needed:

http://sprunge.us/dgsHqH

 

 

Sprint: Execution Team 2020-11-02
Participants:

 Description   

When mongo is running inside a docker container it is impossible to create a new replica set on a fresh config server.

I have tried to find a work around by trying this on versions 4.4.1, 4.7.0, and 4.4.0, all produce the same error. I have also tried running it in a custom built container on Ubuntu 20.04 (focal) and also get the same error.



 Comments   
Comment by Githook User [ 19/Nov/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-51733 fix fromMigrate when creating indexes in the config db

(cherry-picked from commit e9ad50c2eb550e3d1db94ac94ac870ae4c68960e)
Branch: v4.4
https://github.com/mongodb/mongo/commit/4772d113dc17c44b0950540f99a2499eee43e87b

Comment by Githook User [ 19/Nov/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-51733 ShardLocal::createIndexOnConfig() falls back on hybrid index build if collection is not empty

(cherry-picked from commit 6419fd81bdefa91390303fdda48457c8a485399b)
Branch: v4.4
https://github.com/mongodb/mongo/commit/daf7f0886807f3f5e1e0ca0acb9b62cce19225c5

Comment by Githook User [ 19/Nov/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-51733 add test case for ShardLocal::createIndexOnConfig() with non-empty collections

(cherry-picked from commit 3aafc656417787ad650701cc55f3a95dc8e75741)
Branch: v4.4
https://github.com/mongodb/mongo/commit/9d4236bdb3588639fbffcc5bcacc93835bba0925

Comment by Benety Goh [ 28/Oct/20 ]

freemo@gmail.com, thank you for bringing this to our attention!

Separately, we have filed issue #431 in the docker-library/mongo project on GitHub.

Comment by Jeffrey Phillips Freeman [ 28/Oct/20 ]

Thanks so much on the speedy resolve on this one. Great work team!

Comment by Githook User [ 27/Oct/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-51733 fix fromMigrate when creating indexes in the config db
Branch: master
https://github.com/mongodb/mongo/commit/e9ad50c2eb550e3d1db94ac94ac870ae4c68960e

Comment by Githook User [ 27/Oct/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-51733 ShardLocal::createIndexOnConfig() falls back on hybrid index build if collection is not empty
Branch: master
https://github.com/mongodb/mongo/commit/6419fd81bdefa91390303fdda48457c8a485399b

Comment by Githook User [ 27/Oct/20 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-51733 add test case for ShardLocal::createIndexOnConfig() with non-empty collections
Branch: master
https://github.com/mongodb/mongo/commit/3aafc656417787ad650701cc55f3a95dc8e75741

Comment by Jeffrey Phillips Freeman [ 23/Oct/20 ]

I can certainly do that. But given the nature of the error, and the fact that it is running standard mongo executables I would imagine in the end it would still come down to a bug in mongo itself. The key will be replicating it outside the container I suppose.

Comment by Jonathan Streets (Inactive) [ 23/Oct/20 ]

hi freemo@gmail.com, thank you for reporting the issue. We've done some investigation and I can use your reproducer and get the error using the Community Supported docker.com container running on Ubuntu 18, however the commands succeed for both Ubuntu 18, and in the Mongo Supported Enterprise Operator.

The container on docker.com is Community maintained and supported. Please can you report the issue there ?

jon

Comment by Jeffrey Phillips Freeman [ 19/Oct/20 ]

FYI It does appear 3.6.20 however is not affected by this bug, that version seems to work just fine, presumably earlier versions as well.

Generated at Thu Feb 08 05:26:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.