[SERVER-32218] Fatal Assertion 40486 at src/mongo/transport/transport_layer_asio.cpp 170 Created: 08/Dec/17  Updated: 20/Jun/23  Resolved: 08/Dec/17

Status: Closed
Project: Core Server
Component/s: Networking, Upgrade/Downgrade
Affects Version/s: 3.6.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Stefan Bieliauskas [X] Assignee: Jonathan Reams
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-77903 Upgrade from version 4.4.15 to versio... Closed
Operating System: ALL
Steps To Reproduce:

1. Use the latest docker image
2. Add to your dockerfile: RUN mongod --fork --logpath /var/log/mongodbdo.log --dbpath /data/db2 && <Some command to restore data via mongorestore>
3. Add CMD ["mongod", "--config", "/etc/mongodb.conf"]
4. Start the container and get the error

Participants:

 Description   

I've got the following error message by using the latest docker image 3:latest and 3:rc-jessie
I'm grateful for every hint or help that I can get on this issue.

Error Message

08/12/2017 10:16:592017-12-08T09:16:59.967+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db2 64-bit host=4c8fe5e23ea5
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten] db version v3.6.0-rc8
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten] git version: 3c81217d2c14aa335f7273fb822541e31a2cc0f1
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten] allocator: tcmalloc
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten] modules: none
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten] build environment:
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten]     distmod: debian81
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten]     distarch: x86_64
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten]     target_arch: x86_64
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", security: { authorization: "enabled" }, storage: { dbPath: "/data/db2" } }
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 E NETWORK  [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock Operation not permitted
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 F -        [initandlisten] Fatal Assertion 40486 at src/mongo/transport/transport_layer_asio.cpp 170
08/12/2017 10:16:592017-12-08T09:16:59.968+0000 F -        [initandlisten]
08/12/2017 10:16:59



 Comments   
Comment by Ramon Fernandez Marina [ 08/Dec/17 ]

Thanks for sharing your dockerfiles here Stefan11291129, and glad to hear you've found a way forward. As Jonathan mentioned above, another workaround is to disable the unix domain socket altogether if you don't need it.

Closing this as a duplicate of SERVER-32216, which comes with a code change to make improvements in this area.

Regards,
Ramón.

Comment by Stefan Bieliauskas [X] [ 08/Dec/17 ]

Thank you for your quick reply and your comments on this issue.
The hints and explanations were very helpful and I managed to solve the problem and you are fast. Really good support team, thanks!

As you mentioned this problem occurs only if you try to start the database with different users.
Just to clarify the problem more in detail:

  • During the docker build process, I started a mongo server on port 27017 and used this instance to restore the database.
  • After I restored the database I shutdown the instance by killing the process (because I started the mongo instance in fork mode)
  • During the startup of a container, the docker environment seems to be slightly different (same username but indeed a different user)

How to fix it? - Start all mongo instances on a different port with the `--port` argument during the docker build.

Maybe somebody else will help this here the Dockerfile that caused the problem:

FROM mongo:3
 
 
#The mongodb server without port or protocoll like db.rightmart.de
ARG BACKUP_MONGO_HOST
#The mongodb port
ARG BACKUP_MONGO_PORT
#The user with read access to the database you want to backup
ARG BACKUP_MONGO_USER
#The password for the user
ARG BACKUP_MONGO_PASS
#The database name you want to backup
ARG BACKUP_MONGO_DB_NAME
#The password for the database that the container expose (choose randomly
ARG MONGO_DB_PASS
#The user for the database (choose randomly)
ARG MONGO_DB_USER
#The target database for the restore command
ARG MONGO_DB_NAME
 
RUN mongodump  \
    --host $BACKUP_MONGO_HOST \
    --port $BACKUP_MONGO_PORT \
    --username $BACKUP_MONGO_USER  \
    --password $BACKUP_MONGO_PASS \
    --db $BACKUP_MONGO_DB_NAME   \
    --authenticationDatabase admin \
    --out /backup/mongodump-latest
 
# Modify child mongo to use /data/db2 as dbpath (because /data/db wont persist the build)
RUN mkdir -p /data/db2 \
    && echo "dbpath = /data/db2" > /etc/mongodb.conf \
    && chown -R mongodb:mongodb /data/db2
 
 
RUN mongod --fork --logpath /var/log/mongodbdo.log --dbpath /data/db2 \
    && mongorestore  \
    --host localhost \
    --nsFrom $BACKUP_MONGO_DB_NAME"*" \
    --nsTo $MONGO_DB_NAME"*"   /backup/mongodump-latest \
    && mongod --dbpath /data/db2 --shutdown \
    && chown -R mongodb /data/db2
 
RUN mongod --fork --logpath /var/log/mongodbdo.log --dbpath /data/db2 \
    && mongo --eval "db.Webhooks.deleteMany({})" \
    && mongo --eval "printjson(db.adminCommand( { \
                                       createUser: \""$MONGO_DB_USER"\", \
                                       pwd: \""$MONGO_DB_PASS"\", \
                                       roles: [ \
                        
                                        { role: \"readWrite\", db: \""$MONGO_DB_NAME"\" },\
                                        \"readWrite\" \
                                       ] \
                                     }))" \
    && mongod --dbpath /data/db2 --shutdown \
    && chown -R mongodb /data/db2
 
RUN mkdir -p /data/db2 \
    && echo "dbpath = /data/db2 \nauth = true " > /etc/mongodb.conf \
    && cat /etc/mongodb.conf
 
RUN rm -r -f /backup/mongodump-latest
 
VOLUME /data/db2
 
CMD ["mongod", "--config", "/etc/mongodb.conf"]

And here the fixed version with the different port for the internal build process mongo instnace.

FROM mongo:3
 
 
#The mongodb server without port or protocoll like db.rightmart.de
ARG BACKUP_MONGO_HOST
#The mongodb port
ARG BACKUP_MONGO_PORT
#The user with read access to the database you want to backup
ARG BACKUP_MONGO_USER
#The password for the user
ARG BACKUP_MONGO_PASS
#The database name you want to backup
ARG BACKUP_MONGO_DB_NAME
#The password for the database that the container expose (choose randomly
ARG MONGO_DB_PASS
#The user for the database (choose randomly)
ARG MONGO_DB_USER
#The target database for the restore command
ARG MONGO_DB_NAME
 
RUN mongodump  \
    --host $BACKUP_MONGO_HOST \
    --port $BACKUP_MONGO_PORT \
    --username $BACKUP_MONGO_USER  \
    --password $BACKUP_MONGO_PASS \
    --db $BACKUP_MONGO_DB_NAME   \
    --authenticationDatabase admin \
    --out /backup/mongodump-latest
 
# Modify child mongo to use /data/db2 as dbpath (because /data/db wont persist the build)
RUN mkdir -p /data/db2 \
    && echo "dbpath = /data/db2" > /etc/mongodb.conf \
    && chown -R mongodb:mongodb /data/db2
 
ENV MONGO_RESTORE_PORT=27118
 
RUN mongod --port $MONGO_RESTORE_PORT --fork --logpath /var/log/mongodbdo.log --dbpath /data/db2 \
    && mongorestore  \
    --host localhost \
    --port $MONGO_RESTORE_PORT \
    --nsFrom $BACKUP_MONGO_DB_NAME"*" \
    --nsTo $MONGO_DB_NAME"*"   /backup/mongodump-latest \
    && mongod --dbpath /data/db2 --shutdown \
    && chown -R mongodb /data/db2
 
RUN mongod --port $MONGO_RESTORE_PORT --fork --logpath /var/log/mongodbdo.log --dbpath /data/db2 \
    && mongo --port $MONGO_RESTORE_PORT --eval "db.Webhooks.deleteMany({})" \
    && mongo --port $MONGO_RESTORE_PORT --eval "printjson(db.adminCommand( { \
                                       createUser: \""$MONGO_DB_USER"\", \
                                       pwd: \""$MONGO_DB_PASS"\", \
                                       roles: [ \
                                        { role: \"readWrite\", db: \""$MONGO_DB_NAME"\" },\
                                        \"readWrite\" \
                                       ] \
                                     }))" \
    && mongod --port $MONGO_RESTORE_PORT --dbpath /data/db2 --shutdown \
    && chown -R mongodb /data/db2
 
RUN mkdir -p /data/db2 \
    && echo "dbpath = /data/db2 \nauth = true " > /etc/mongodb.conf \
    && cat /etc/mongodb.conf
 
RUN rm -r -f /backup/mongodump-latest
 
VOLUME /data/db2
 
CMD ["mongod", "--config", "/etc/mongodb.conf"]

Comment by Jonathan Reams [ 08/Dec/17 ]

3.6 has a new networking implementation, but they should both have the same behaviour here. Could you try running the mongod with --transportLayer=legacy to see if the problem reproduces? That should switch mongodb back to using the 3.4 networking code. You can also work around this problem by disabling UNIX sockets if you aren't using them (see https://docs.mongodb.com/manual/reference/configuration-options/#net-unixdomainsocket-options)

Comment by Ramon Fernandez Marina [ 08/Dec/17 ]

I can reproduce this error stand-alone if I run a mongod on port NNNN, then kill -9 that mongod, and then try to start another mongod on the same port as a different user.

I'm not very familiar with docker, but reading the commands above makes me wonder whether you're trying to run two mongod processes as the same user on the same port, which would explain this issue. Stefan11291129, can you please confirm whether this is the case?

Regards,
Ramón.

PS:
Please note that MongoDB does not produce an "official" Docker image yet, but we'll check whether there are issues with its packaging.

Comment by Stefan Bieliauskas [X] [ 08/Dec/17 ]

Please close thus issue

Comment by Stefan Bieliauskas [X] [ 08/Dec/17 ]

Problem solved by adding:

RUN rm /tmp/mongodb-27017.sock

to my dockerfile

Generated at Thu Feb 08 04:29:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.