[SERVER-77903] Upgrade from version 4.4.15 to version 6.0.6 Created: 07/Jun/23  Updated: 06/Dec/23  Resolved: 06/Dec/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Полимед Полимед Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: Ubuntu 20.04.4 LTS


Issue Links:
Related
is related to SERVER-32216 With Mongo 3.6(in docker) we hit this... Closed
is related to SERVER-32218 Fatal Assertion 40486 at src/mongo/tr... Closed
Participants:

 Description   

Problem Statement/Rationale

What is going wrong? What action would you like the Engineering team to take?

Please be sure to attach relevant logs with any sensitive data redacted.
How to retrieve logs for: Compass; Shell

Steps to Reproduce

  1. rs01:PRIMARY> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1
  2. I did all the steps described here: https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/
  3. systemctl daemon-reload
  4. service mongod start
  5. service mongod status
  6.  mongod.service - MongoDB Database Server
         Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
         Active: failed (Result: exit-code) since Wed 2023-06-07 15:39:32 UTC; 3s ago
           Docs: https://docs.mongodb.org/manual
        Process: 15800 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=14)
       Main PID: 15800 (code=exited, status=14)
            CPU: 58ms

Jun 07 15:39:32 copy-chat systemd[1]: Started MongoDB Database Server.
Jun 07 15:39:32 copy-chat mongod[15800]: {"t":

{"$date":"2023-06-07T15:39:32.682Z"}

,"s":"I",  "c":"CONTROL",  "id":7484500, "ctx":"-","msg":"Environment variable MONGODB_CONFIG_OVERRIDE_NOFORK == 1, overridi>
Jun 07 15:39:32 copy-chat systemd[1]: mongod.service: Main process exited, code=exited, status=14/n/a
Jun 07 15:39:32 copy-chat systemd[1]: mongod.service: Failed with result 'exit-code'.

 

 

/var/log/mongodb/mongod.log

{"t":\{"$date":"2023-06-07T15:41:23.211+00:00"}

,"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"-","msg":"***** SERVER RESTARTED *****"}

{"t":\{"$date":"2023-06-07T15:41:23.218+00:00"}

,"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}

{"t":\{"$date":"2023-06-07T15:41:23.218+00:00"}

,"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWi>

{"t":\{"$date":"2023-06-07T15:41:23.218+00:00"}

,"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenC>

{"t":\{"$date":"2023-06-07T15:41:23.277+00:00"}

,"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namesp>

{"t":\{"$date":"2023-06-07T15:41:23.277+00:00"}

,"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","na>

{"t":\{"$date":"2023-06-07T15:41:23.277+00:00"}

,"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","namespace":>

{"t":\{"$date":"2023-06-07T15:41:23.277+00:00"}

,"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}

{"t":\{"$date":"2023-06-07T15:41:23.278+00:00"}

,"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":15838,"port":27017,"dbPath":"/var/lib/mongodb","architectu>

{"t":\{"$date":"2023-06-07T15:41:23.278+00:00"}

,"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"6.0.6","gitVersion":"26b4851a412cc8b9b4a18cdb6>

{"t":\{"$date":"2023-06-07T15:41:23.278+00:00"}

,"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":

{"name":"Ubuntu","version":"20.04"}

}}

{"t":\{"$date":"2023-06-07T15:41:23.278+00:00"}

,"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp>

{"t":\{"$date":"2023-06-07T15:41:23.279+00:00"}

,"s":"E",  "c":"NETWORK",  "id":23024,   "ctx":"initandlisten","msg":"Failed to unlink socket file","attr":{"path":"/tmp/mongodb-27017.sock","error":"Operation >

{"t":\{"$date":"2023-06-07T15:41:23.279+00:00"}

,"s":"F",  "c":"ASSERT",   "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":40486,"file":"src/mongo/transport/transport_layer_asio.cp>

{"t":\{"$date":"2023-06-07T15:41:23.279+00:00"}

,"s":"F",  "c":"ASSERT",   "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

Expected Results

What do you expect to happen?

Actual Results

What do you observe is happening?

Additional Notes

Any additional information that may be useful to include.



 Comments   
Comment by Chris Kelly [ 06/Dec/23 ]

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Comment by Chris Kelly [ 11/Jul/23 ]

Just checking in!

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested information?

Comment by Chris Kelly [ 20/Jun/23 ]

Hi  project@polimedad.com,

Thanks for your report. Some notes:

  • Failed to unlink socket file would indicate a permissions issue to me, at first glance. Other tickets have covered this issue and I would invite you to check out these and see if they resolve your issue:
    • SERVER-32218 / SERVER-32216 - Fatal Assertion 40486 at src/mongo/transport/transport_layer_asio.cpp
      • This was reproducible by running a mongod on port NNNN, then killing that mongod, and then trying to start another mongod on the same port as a different user.
      • As mentioned in those tickets, a possible workaround is to disable the unix domain socket altogether if you don't need it.
      • Make sure mongodb has the appropriate permissions to read/write the file (see here)
  • Mongodb does not condone skipping major versions when upgrading - the supported method is to upgrade to every major release inbetween. In your case, you should be upgrading from 4.4 to 5.0, then 6.0
  • mmapv1 has been removed as of 4.2

 

If the above does not resolve your issue, please summarize a more specific timeline of the exact events leading to the issue, and submit the full logs. For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to the ticket:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)

Christopher

Comment by Полимед Полимед [ 08/Jun/23 ]

 cat /etc/mongod.conf

  1. mongod.conf
  1. for documentation of all options, see:
  2.   http://docs.mongodb.org/manual/reference/configuration-options/
  1. Where and how to store data.
    storage:
      dbPath: /var/lib/mongodb
      journal:
        enabled: true
  2.  engine:
  3.  mmapv1:
  4.  wiredTiger:
  1. where to write logging data.
    systemLog:
      destination: file
      logAppend: true
      path: /var/log/mongodb/mongod.log
  1. network interfaces
    net:
      port: 27017
      bindIp: 127.0.0.1
  1. how the process runs
    processManagement:
      timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:

replication:
  replSetName: rs01

#sharding:

    1. Enterprise-Only Options:

#auditLog:

#snmp:

 

 mongod --version

db version v6.0.6
Build Info: {
    "version": "6.0.6",
    "gitVersion": "26b4851a412cc8b9b4a18cdb6cd0f9f642e06aa7",
    "openSSLVersion": "OpenSSL 1.1.1f  31 Mar 2020",
    "modules": [],
    "allocator": "tcmalloc",
    "environment":

{         "distmod": "ubuntu2004",         "distarch": "x86_64",         "target_arch": "x86_64"     }

}

syslog

Jun  8 08:10:55 copy-chat systemd[1]: Started MongoDB Database Server.
Jun  8 08:10:55 copy-chat mongod[16582]: {"t":

{"$date":"2023-06-08T08:10:55.193Z"}

,"s":"I",  "c":"CONTROL",  "id":7484500, "ctx":"-","msg":                                                                    "Environment variable MONGODB_CONFIG_OVERRIDE_NOFORK == 1, overriding \"processManagement.fork\" to false"}
Jun  8 08:10:55 copy-chat systemd[1]: mongod.service: Main process exited, code=exited, status=14/n/a
Jun  8 08:10:55 copy-chat systemd[1]: mongod.service: Failed with result 'exit-code'.
Jun  8 08:10:56 copy-chat rocketchat[129]: {"level":50,"time":"2023-06-08T08:10:56.373Z","pid":129,"hostname":"copy-chat","name":"System","                                                                    msg":"No real time data received recently"}
Jun  8 08:11:00 copy-chat rocketchat[129]: MongoServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017
Jun  8 08:11:00 copy-chat rocketchat[129]:     at Timeout._onTimeout (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/no                                                                    de_modules/mongodb/lib/sdam/topology.js:292:38)
Jun  8 08:11:00 copy-chat rocketchat[129]:     at listOnTimeout (internal/timers.js:557:17)
Jun  8 08:11:00 copy-chat rocketchat[129]:     at processTimers (internal/timers.js:500:7) {
Jun  8 08:11:00 copy-chat rocketchat[129]:   reason: TopologyDescription

{ Jun  8 08:11:00 copy-chat rocketchat[129]:     type: 'ReplicaSetNoPrimary', Jun  8 08:11:00 copy-chat rocketchat[129]:     servers: Map(1) \{ '127.0.0.1:27017' => [ServerDescription] }

,
Jun  8 08:11:00 copy-chat rocketchat[129]:     stale: false,
Jun  8 08:11:00 copy-chat rocketchat[129]:     compatible: true,
Jun  8 08:11:00 copy-chat rocketchat[129]:     heartbeatFrequencyMS: 10000,
Jun  8 08:11:00 copy-chat rocketchat[129]:     localThresholdMS: 15,

Comment by PM Bot [ 07/Jun/23 ]

Hello project@polimedad.com, thank you for reaching out to us! The team will review your issue and get back to you soon as soon as possible.

Please review your issue to ensure you've included your environment details and have attached relevant logs (with any sensitive data redacted), so that we're best able to provide you a timely and thorough response. Thanks again!

Generated at Thu Feb 08 06:36:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.