[SERVER-38447] create index blocked by checkpoint mutex Created: 07/Dec/18 Updated: 07/Apr/23 Resolved: 01/Nov/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Concurrency |
| Affects Version/s: | 4.0.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | chensi | Assignee: | Backlog - Triage Team |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Server Triage
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
when create index with background on one database, the database x lock held for a long time, with the help of gdb, i found it was blocked by the wiredtiger's checkpoint mutex, the gdb stack is: Thread 145 (Thread 0x7fab66ff4700 (LWP 1426590)): |
| Comments |
| Comment by Kelsey Schubert [ 01/Nov/21 ] |
|
I believe this issue has been resolved by Thank you, |
| Comment by deyukong [ 27/Dec/18 ] |
|
After a short-time conversation with Jack, we believe the problem has already been described with the stack in @chensi_04@126.com's second post. Mean no offence but no more logs are needed for this problem. We can discuss about it with the latest mongo code base.
1)in commands/create_indexes.cpp, line 311, target db's X lock is held. 2)And in commmands/create_indexes.cpp, line 383, indexer.init is called. If we track the call path, we can find WiredTigerUtil::setTableLogging is called, track deeper, we can find it gets into wiredtiger, session_api.c, line 670, the checkpoint mutex is taken. So, in conclusion, the db's X lock may block for quite a long time if a checkpoint happens to be doing. It may block for several seconds or tens of seconds if the mongod instance is suffering a heavy workload. It may or may not be a bug, but it is caused by the checkpoint mutex. And, what's important is that it's unacceptable for a background index building to block the whole db for quite a long time, because, it is a newly introduced problem. for the compatibilty with old versions, it should be treated as a bug.
|
| Comment by Danny Hatcher (Inactive) [ 10/Dec/18 ] |
|
Hello Jack, Unfortunately, it will be very difficult for us to validate this issue unless we have the logs. I have generated a Secure Upload Portal that you can use to upload files. Only MongoDB engineers can access the portal and files within are deleted after 180 days. Would this satisfy your security concerns? Thank you, Danny |
| Comment by chensi [ 08/Dec/18 ] |
|
1. I send the create index command, but 2. On the primary |
| Comment by Danny Hatcher (Inactive) [ 07/Dec/18 ] |
|
Hello Jack, I cannot validate what will happen if you manually change any code and we cannot troubleshoot any such changes. Can you please provide more context on the issue? Thank you, Danny |
| Comment by chensi [ 07/Dec/18 ] |
|
also, we need to fix this problem as soon as possible, is it ok if i just remove the setTableLogging from the following code: - version too new for this mongod." // Index data format 6 and 9 correspond to KeyString version V0 and data format 8 and 10 if (!isReadOnly) { |
| Comment by chensi [ 07/Dec/18 ] |
|
and, i see this: https://jira.mongodb.org/browse/SERVER-35625 setTableLogging will be removed, but please tell the roadmap, thx. |