[SERVER-74688] Error after upgrade to MongoDB v5.0.14 Created: 08/Mar/23 Updated: 29/Sep/23 Resolved: 27/Jun/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 5.0.14 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Yook Choo | Assignee: | Varun Ravichandran |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Server Security
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Sprint: | Security 2023-05-15, Security 2023-05-29, Security 2023-06-12, Security 2023-06-26, Security 2023-07-10 | ||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
Getting this error in mongod.log every 5 min after upgrade replica set to v5.0.14 {"t":\{"$date":"2023-03-06T08:55:29.323-05:00"},"s":"W", "c":"ACCESS", "id":5626700, "ctx":"conn80","msg":"Client has attempted to reauthenticate as a single user","attr":{"user": {"user":"__system","db":"local"}}}
Replica set seems to be ok, all members synced up. Warning message appeared to be cosmetic, but the error show up in mongod.log every 5 min. I need help finding a solution for this issue.
Sally Choo |
| Comments |
| Comment by Varun Ravichandran [ 23/May/23 ] |
|
Hi sally.choo@shaw.ca , I'm sorry for the delay in responding and thanks for your patience! We added this log statement in the 5.0 release of the server, which is why you did not see it previously in 4.4. This particular instance of the log appears to be occurring because the secondary nodes are using already-authenticated connections to the primary node to perform certain operations. It is not a sign of an error and should not affect your replica set in any way. That being said, we can produce a fix for this on the master branch to prevent this reauthentication from happening and thereby also stop the log from being emitted. See SERVER-78466 to track that fix. I hope this is useful! |
| Comment by Yuan Fang [ 24/Apr/23 ] |
|
I apologize for the late response! thank you for providing the diagnostic dataset. I've passed this along to the team for further investigation. Please stay tuned for further updates. Regards, |
| Comment by Yook Choo [ 10/Mar/23 ] |
|
Hi Yuan Fang, Thanks ! Sally |
| Comment by Yuan Fang [ 08/Mar/23 ] |
|
sally.choo@shaw.ca, I confirm that the uploads can now be seen. Thank you! |
| Comment by Yook Choo [ 08/Mar/23 ] |
|
I just tried the upload again. [root@preprod01-upgtest-mongodb-server83 tmp]# curl -X POST https://upload.box.com/api/2.0/files/content \ [root@preprod01-upgtest-mongodb-server84 tmp]# curl -X POST https://upload.box.com/api/2.0/files/content \ [root@preprod01-upgtest-mongodb-server85 tmp]# curl -X POST https://upload.box.com/api/2.0/files/content \ |
| Comment by Yuan Fang [ 08/Mar/23 ] |
|
Unfortunately, I couldn't find any files in the folder linked to the upload portal. Could you try to upload them to the upload portal again? Please make sure that the files are fully uploaded and you should see the progress after initiating the command. Additionally, it's important to note that providing the full path of the file may not work as intended. Therefore, make sure to navigate to the current location of the file and replace <filename> with only the file name, without including the preceding path. Feel free to let me know if you encounter any issues while uploading. Regards, |
| Comment by Yook Choo [ 08/Mar/23 ] |
|
Hi Yuan Fang, Thanks for the info. I had uploaded the requested logs. These are the logs I uploaded: server83.logs.tar server84.logs.tar server85.logs.tar Here are more detail for you: 3 nodes replica set: preprod01-upgtest-mongodb-server83/84/85 IP: 10.14.131.83/84/85 I upgraded the replica from MongoDB v4.4.16 to v5.0.14 Then I tested the download back to MongoDB v4.4.18 Then I upgrade again to MongoDB v5.0.14 The warning messages only show up on the primary node and only when MongoDB running v5.0.14 When I download the version back to 4.4.18, no warning messages in mongod.log at all When I upgrade to v5.0.14 again the 2nd time, warning messages return No issue with the replica set, all members are synced up at all time, no oplog lagging that I can see. When I step down the primary, the warning message show up on the new primary Thanks ! |
| Comment by Yuan Fang [ 08/Mar/23 ] |
|
Thank you for reporting the issue. Based on the information provided, it appears that the log message is being triggered by the MongoDB automation agent, which is responsible for querying the health of the cluster and performing configuration changes, among other tasks. However, without additional context or information, it is difficult to confirm that. I've created a secure upload portal for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time. For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:
To answer the question of why the log message appeared after upgrading to v5.0.14, could you also specify from which version it was upgraded to v5.0.14? Regards, |