[SERVER-31869] mongodb crashes on package upgrade on Xenial Created: 08/Nov/17  Updated: 07/Jan/18  Resolved: 08/Dec/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.17
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Fred Leeflang Assignee: Mark Agarunov
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File metrics.2017-11-08T05-27-59Z-00000     File mongod.conf     Text File mongod.log    
Operating System: ALL
Participants:

 Description   

After a package upgrade on mongo2, mongodb crashes on startup:

2017-11-08T06:55:48.363+0100 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-11-08T06:55:48.363+0100 I NETWORK [initandlisten] waiting for connections on port 27017
2017-11-08T06:55:48.363+0100 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2017-11-08T06:55:48.363+0100 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2017-11-08T06:55:48.364+0100 I - [signalProcessingThread] Fatal Assertion 18823
2017-11-08T06:55:48.364+0100 I - [signalProcessingThread]
***aborting after fassert() failure
2017-11-08T06:55:48.374+0100 F - [signalProcessingThread] Got signal: 6 (Aborted).
0x1556f12 0x1555eb9 0x1556722 0x7f52c1e67390 0x7f52c1ac1428 0x7f52c1ac302a 0x14d30d3 0x106285a 0xdb9c22 0x1555742 0x1d008b0 0x7f52c1e5d6ba 0x7f52c1b933dd
----- BEGIN BACKTRACE -----
{"backtrace":[
{"b":"400000","o":"1156F12","s":"_ZN5mongo15printStackTraceERSo"}
,
{"b":"400000","o":"1155EB9"}
,
{"b":"400000","o":"1156722"}
,
{"b":"7F52C1E56000","o":"11390"}
,
{"b":"7F52C1A8C000","o":"35428","s":"gsignal"}
,
{"b":"7F52C1A8C000","o":"3702A","s":"abort"}
,
{"b":"400000","o":"10D30D3","s":"_ZN5mongo13fassertFailedEi"}
,
{"b":"400000","o":"C6285A","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl8shutdownEPNS_16OperationContextE"}
,
{"b":"400000","o":"9B9C22","s":"_ZN5mongo11exitCleanlyENS_8ExitCodeE"}
,
{"b":"400000","o":"1155742"}
,
{"b":"400000","o":"19008B0"}
,
{"b":"7F52C1E56000","o":"76BA"}
,
{"b":"7F52C1A8C000","o":"1073DD","s":"clone"}
],"processInfo":{ "mongodbVersion" : "3.2.17", "gitVersion" : "186656d79574f7dfe0831a7e7821292ab380f667", "compiledModules" : [], "uname" :
{ "sysname" : "Linux", "release" : "4.4.0-98-generic", "version" : "#121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017", "machine" : "x86_64" }
, "somap" : [
{ "elfType" : 2, "b" : "400000", "buildId" : "7B0295168DF58E0C8E08A92D04CA518C8F9B3C86" }
,
{ "b" : "7FFFD67EF000", "elfType" : 3, "buildId" : "30999EF567346F2D006F2EE03E2A4D94CD144519" }
,
{ "b" : "7F52C2DE2000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "42D1D917636C64CF990465F087B181697E4CDD7F" }
,
{ "b" : "7F52C299E000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "C09D2E075184E1CA71F7E08957982A947A377939" }
,
{ "b" : "7F52C2796000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }
,
{ "b" : "7F52C2592000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }
,
{ "b" : "7F52C2289000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }
,
{ "b" : "7F52C2073000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }
,
{ "b" : "7F52C1E56000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }
,
{ "b" : "7F52C1A8C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }
,
{ "b" : "7F52C304B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }
] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1556f12]
mongod(+0x1155EB9) [0x1555eb9]
mongod(+0x1156722) [0x1556722]
libpthread.so.0(+0x11390) [0x7f52c1e67390]
libc.so.6(gsignal+0x38) [0x7f52c1ac1428]
libc.so.6(abort+0x16A) [0x7f52c1ac302a]
mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14d30d3]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl8shutdownEPNS_16OperationContextE+0x18A) [0x106285a]
mongod(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x92) [0xdb9c22]
mongod(+0x1155742) [0x1555742]
mongod(+0x19008B0) [0x1d008b0]
libpthread.so.0(+0x76BA) [0x7f52c1e5d6ba]
libc.so.6(clone+0x6D) [0x7f52c1b933dd]



 Comments   
Comment by Fred Leeflang [ 08/Dec/17 ]

Sorry about that Mark, I no longer work for that company.

Thanks for your time though,
Fred

Comment by Mark Agarunov [ 08/Dec/17 ]

Hello fredl

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Thanks,
Mark

Comment by Mark Agarunov [ 01/Dec/17 ]

Hello fredl,

We still need additional information to diagnose the problem. If this is still an issue for you, could you please provide the log files from the affected nodes?

Thanks,
Mark

Comment by Mark Agarunov [ 09/Nov/17 ]

Hello fredl,

Thank you for the information. I'll set this ticket to "Waiting for User Input" until you have a chance to get this data.

Thanks,
Mark

Comment by Fred Leeflang [ 09/Nov/17 ]

Hi Mark,

I will discuss with our team if this is possible over the weekend. These are production servers. It should not take more than a few minutes from the looks of it so I don't expect it to be a big issue.

Thanks,
Fred

Comment by Mark Agarunov [ 08/Nov/17 ]

Hello fredl,

Thank you for the report. To get a better idea of what may be causing this behavior, could you please set the log verbosity to 5, attempt to start mongod with systemd, and provide the complete log file? If possible, please also provide systemd's log output via journalctl -u mongodb > mongodb.service.log. This should five some insight into why mongodb is not starting from systemd.

Thanks,
Mark

Comment by Fred Leeflang [ 08/Nov/17 ]

ARGH. ARGH.

This is apparently caused by trying to start it through systemd.

I finally ended up typing simply:

root@mongo2 /var/log/mongodb # sudo -H -u mongodb bash -c "/usr/bin/mongod -f /etc/mongod.conf"

Which works fine.

HOW CAN THIS BE????

Comment by Fred Leeflang [ 08/Nov/17 ]

I see that bug has been closed but don't see the resolution

Comment by Fred Leeflang [ 08/Nov/17 ]

Looks like it may be this bug: https://jira.mongodb.org/browse/SERVER-29850

Comment by Fred Leeflang [ 08/Nov/17 ]

I tried switching the user in the mongodb.service file to root again but output in log remains the same.

Comment by Fred Leeflang [ 08/Nov/17 ]

The last lines in the mongod.log make it seem like it's running, however it's not.

Comment by Fred Leeflang [ 08/Nov/17 ]

Another difference in the mongodb.service file was that the old one used User=root, something I want to prevent at all costs but which is probably the big showstopper for now.

There were a few files in /var/lib/mongodb that were owned by root, I changed these all to mongodb:mongodb but that didn't fix the problem

Comment by Fred Leeflang [ 08/Nov/17 ]

So I made the mongodb.service file look like:

[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target

[Service]

  1. (file size)
    LimitFSIZE=infinity
  2. (cpu time)
    LimitCPU=infinity
  3. (virtual memory size)
    LimitAS=infinity
  4. (open files)
    LimitNOFILE=64000
  5. (processes/threads)
    LimitNPROC=64000
    User=mongodb
    ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf

[Install]
WantedBy=multi-user.target

Comment by Fred Leeflang [ 08/Nov/17 ]

The big difference is that yesterday I when it was still running there was another /etc/systemd/system/mongodb.service file. I made this new one from instructions on https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04

This one didn't have:

[Service]

  1. (file size)
    LimitFSIZE=infinity
  2. (cpu time)
    LimitCPU=infinity
  3. (virtual memory size)
    LimitAS=infinity
  4. (open files)
    LimitNOFILE=64000
  5. (processes/threads)
    LimitNPROC=64000
Generated at Thu Feb 08 04:28:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.