[SERVER-75994] "Collection refresh failed": No chunks were found for the collection config.system.sessions Created: 12/Apr/23  Updated: 19/May/23  Resolved: 19/May/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ulan Kospanov Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: Bug
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
backports SERVER-75995 "Collection refresh failed": No chunk... Closed
Duplicate
is duplicated by SERVER-75996 "Collection refresh failed": No chunk... Closed
Related
related to SERVER-75995 "Collection refresh failed": No chunk... Closed
Operating System: ALL
Participants:
Story Points: 5

 Description   

We sharded our db, after that, there is problem with Logical Session Cache Refresh, because, in logs we can see such as lines:

"ctx":"LogicalSessionCacheRefresh","msg":"Collection refresh failed","attr":{"namespace":"config.system.sessions","exception":"ConflictingOperationInProgress: No chunks were found for the collection config.system.sessions"}}

What is already done:
1. setFeatureCompatibilityVersion: already "5.0";
2. We tried drop collection and then recreate, but "NamespaceNotFound" error when i run drop command, even in config server.

MongoDB version "5.0.4".

 



 Comments   
Comment by Chris Kelly [ 19/May/23 ]

kospanovulan@gmail.com,

Thanks for adding the extra context! I'll go ahead and close this for now.

Comment by Ulan Kospanov [ 17/May/23 ]

Hi, chris.kelly@mongodb.com , thank you, We resolved the issue. In my case, there are some steps:
1. Update all shards to 5.0.16;
2. db.system.sessions.drop() in primary config shard server;
3. Delete document with {"_id": "config.system.sessions"} in config.collections;
4. Then, in the next iteration of Cache Updating, MongoDB with version >5.0.16 will automatically create new collection "config.system.sessions" and shard it. 

That's all. But, I recommend, do this steps only as a last resort, because MongoDB developers dont't recommend manually change config database)

Good luck!

Comment by Ulan Kospanov [ 17/May/23 ]

More new log messages after updating

{"t":{"$date":"2023-05-17T10:10:05.227+06:00"},"s":"I",  "c":"SHARDING", "id":22650,   "ctx":"replSetDistLockPinger","msg":"Unlocked distributed lock","attr":{"lockSessionId":{"$oid":"010000000000000000000000"},"lockName":"config"}}
{"t":{"$date":"2023-05-17T10:10:05.228+06:00"},"s":"I",  "c":"CONTROL",  "id":20714,   "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"IllegalOperation: Failed to create config.system.sessions :: caused by :: collections in the config db must be empty to be sharded"}}
 

Comment by Ulan Kospanov [ 17/May/23 ]

In the logs, there is new error messages.

{"t":{"$date":"2023-05-17T09:53:22.565+06:00"},"s":"I",  "c":"SH_REFR",  "id":4086500, "ctx":"LogicalSessionCacheReap","msg":"Collection refresh failed","attr":{"namespace":"config.system.sessions","exception":"ConflictingOperationInProgress: No chunks were found for the collection config.system.sessions"}}
{"t":{"$date":"2023-05-17T09:53:22.565+06:00"},"s":"I",  "c":"CONTROL",  "id":20712,   "ctx":"LogicalSessionCacheReap","msg":"Sessions collection is not set up; waiting until next sessions reap interval","attr":{"error":"ConflictingOperationInProgress: No chunks were found for the collection config.system.sessions"}}
{"t":{"$date":"2023-05-17T09:53:22.567+06:00"},"s":"I",  "c":"CONTROL",  "id":20714,   "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"NotWritablePrimary: Failed to create config.system.sessions :: caused by :: Not primary while running findAndModify command on collection config.locks"}}
{"t":{"$date":"2023-05-17T09:53:22.568+06:00"},"s":"W",  "c":"SHARDING", "id":22670,   "ctx":"replSetDistLockPinger","msg":"Error unlocking distributed lock","attr":{"lockName":"config","lockSessionId":{"$oid":"010000000000000000000000"},"error":"NotWritablePrimary: Not primary while running findAndModify command on collection config.locks"}} 

Comment by Ulan Kospanov [ 05/May/23 ]

Hi, chris.kelly@mongodb.com ,

We are going to update our mongoDB to 6.0v. As soon as we finish, I'll write here according to the results)

Thanks for reply)

Comment by Chris Kelly [ 04/May/23 ]

Hi kospanovulan@gmail.com

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested info above?

Comment by Chris Kelly [ 12/Apr/23 ]

Hi kospanovulan@gmail.com 

Some initial thoughts:

It is not recommended to modify these collections. If they were modified manually, it would probably make sense that it impacts logical session cache refresh on mongos which are reaching out to it.

You mention FCV is set to 5.0, did you recently upgrade from 4.4 around when this was sharded (before or after?)

It's worth mentioning that there have been a number of minor version updates since 5.0.4, and you may find this issue doesn't exist on the newest patch release of 5.0 (currently 5.0.16), including 

  • SERVER-57662 - Wait for config.system.sessions collection to exist on the config server before refreshing logical session cache (5.0.7)
  • SERVER-61214 - Ensure having the latest known entry of the catalog cache when creating config.system.sessions (5.0.6)

I would consider trying it out on the latest patch release to confirm the above (or any other fixed issues) aren't contributing to your problem.

If you find the above hasn't resolved the issue:

I've created a secure upload portal for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)

 

Comment by Ulan Kospanov [ 12/Apr/23 ]

my email: kospanovulan@gmail.com

Generated at Thu Feb 08 06:31:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.