[SERVER-72817] On initial run mongod ignores replSet option set in cli Created: 13/Jan/23  Updated: 17/Jan/23  Resolved: 17/Jan/23

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ivan Cherviakov Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

kubernetes cluster (k3s) installed in AWS ec2
image: mongo:4.4.4-bionic


Operating System: ALL
Participants:

 Description   

TL DR
mongod --wiredTigerCacheSizeGB 0.25 --replSet rs0 --keyFile /data/db/security.keyFile ignored replSet param on initial run (means there no /data/db folder yet) in docker container, as running in js script rs.initiate() command showing following non-fatal error in mongod logs
```
MongoDB server version: 4.4.4

{     "ok" : 0,     "errmsg" : "This node was not started with the replSet option",     "code" : 76,     "codeName" : "NoReplicationEnabled" }

```
Also mongod logs showing following for cli params passed
```
Options set by command line","attr":{"options":{"net":{"bindIp":"127.0.0.1","port":27017,"tls":{"mode":"disabled"}},"processManagement":{"fork":true,"pidFilePath":"/tmp/docker-entrypoint-temp-mongod.pid"},"security":{"keyFile":"/data/db/security.keyFile"},"storage":{"wiredTiger":{"engineConfig":

{"cacheSizeGB":0.25}

}},"systemLog":{"destination":"file","logAppend":true,"path":"/proc/1/fd/1"}}}}
```

Using following manifests:
```
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    app: mongo
spec:
  ports:
  - port: 27017
  selector:
    app: mongo

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo
  namespace: default
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
        tier: base
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: mongo
        image: mongo:4.4.4-bionic
        args:
          - "--wiredTigerCacheSizeGB"
          - "0.25"
          - "--replSet"
          - "rs0"
          - "--keyFile"
          - "/data/db/security.keyFile"
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: data
          mountPath: /data/db
        - name: entrypoint
          mountPath: /docker-entrypoint-initdb.d/
        env:
          - name: MONGO_INITDB_DATABASE
            valueFrom:
              configMapKeyRef:
                key: MONGO_DB
                name: envars
          - name: MONGO_INITDB_ROOT_USERNAME
            valueFrom:
              secretKeyRef:
                key: MONGO_USER
                name: sealed-secrets
          - name: MONGO_INITDB_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                key: MONGO_PASS
                name: sealed-secrets
      volumes:
      - name: data
        hostPath:
          path: /data/mongodata
          type: DirectoryOrCreate
      - name: entrypoint
        secret:
          secretName: mongo-init
```
mongo-init secret value is
#/bin/bash

sleep 10s
mongo -u ${MONGO_INITDB_ROOT_USERNAME} -p ${MONGO_INITDB_ROOT_PASSWORD} <<EOF
rs.initiate()
EOF

sleep 10s



 Comments   
Comment by Chris Kelly [ 17/Jan/23 ]

Thanks for sharing additional context on this so others can benefit! Yeah, this component has been maintained by the docker community.

It's not immediately clear to me why this is being parsed out intentionally, but modifying the entrypoint script should work in your case (or just working with the restrictions in place through the means you outlined). This would be a good point to bring up for discussion in their repo https://github.com/docker-library/mongo to clarify this.

I'm going to go ahead and close the ticket since this is not currently a SERVER issue. Thanks for the report!

Christopher

Comment by Ivan Cherviakov [ 13/Jan/23 ]

Checking this file in docker mongo repository, I can see that it is basically wrapper over running mongod, while some arguments are hardcoded and some others (like replSet) are removed under various conditions. Also number of additional actions and checks performed there, which not happen when you start mongod outside of the docker.

Not sure why it is made this way, perhaps there securty considerations? Anyway, with given setup one of consequences is that replica set initiation will be working properly only in 1 case: when you initiate it in 1 node and then you not really mind localhost constraints, which is not what users would do. And therefore it all makes sense and not much can be improved here, aside of just documenting it on docker side.

 

 

Comment by Ivan Cherviakov [ 13/Jan/23 ]

Thanks @Chris Kelly for checking this. In given github issues I can see comment saying that --replSet param should never be needed during initialization, though I believe (and already asked) that behaviour inside docker should be consistent with behaviour outside of it, no matter of which case may not be important.

So far it indeed seems to be issue on docker side, not mongodb server.

Comment by Ivan Cherviakov [ 13/Jan/23 ]

Also this SERVER RESTARTED log is particularry confusing in this case
```

about to fork child process, waiting until server is ready for connections.
forked process: 37

{"t":\{"$date":"2023-01-13T06:52:29.941+00:00"}

,"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"main","msg":"***** SERVER RESTARTED *****"}
```

Comment by Chris Kelly [ 13/Jan/23 ]

Hi ivan.c@taskworld.com,

Note that the mongo Docker image is maintained by the Docker community. As such, we may not be able to provide much input here.

It's bizarre that you don't even see the option in:

 

Options set by command line","attr":{"options":{"net":{"bindIp":"127.0.0.1","port":27017,"tls":{"mode":"disabled"}},"processManagement":{"fork":true,"pidFilePath":"/tmp/docker-entrypoint-temp-mongod.pid"},"security":{"keyFile":"/data/db/security.keyFile"},"storage":{"wiredTiger":{"engineConfig":
{"cacheSizeGB":0.25}

Potentially related: https://github.com/docker-library/mongo/issues/354 

I was able to test this out independently and found that you should see the option on 4.4.4 if you're passing it in to the mongod without a docker image on an initial run. So, this looks like something else weird might be happening to make the option not show up for you. For this issue we'd like to encourage you to start by asking our community for help by posting on the MongoDB Developer Community Forums. If the discussion there leads you to suspect a bug in the MongoDB server, then we'd want to investigate it as a possible bug here in the SERVER project.

Christopher

Comment by Ivan Cherviakov [ 13/Jan/23 ]

So there was another deployment, where script executing rs.initate() worked normally. After checking it appears that script executed after mongodb finished its initialization (setup /data/db), and started with parameters passed. As we can see from the logs, initialization part is mongod process fork itself, do stuff needed and then kill the child, then starting normally. We just need to wait a bit more then.

I wonder if we can programmatically check in the script whether mongodb server currently is in initialization state (that forked process) or already started?

Generated at Thu Feb 08 06:22:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.