[SERVER-30790] ServerStatus on WiredTiger accesses the storage engine without any locks Created: 23/Aug/17 Updated: 30/Oct/23 Resolved: 30/Aug/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | 3.4.9, 3.5.13 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Daniel Gottlieb (Inactive) | Assignee: | Daniel Gottlieb (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Backport Requested: |
v3.4
|
||||||||||||||||
| Sprint: | Storage 2017-09-11 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||
| Description |
|
Note that "ServerStatus" is also called by FTDC which isn't necessarily shutdown before the storage engine shutsdown. Moreover, recover to a timestamp expects the global lock to be sufficient for requiring the system to keep its dirty paws from accessing the storage engine during this critical and complicated operation. |
| Comments |
| Comment by Githook User [ 01/Sep/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'username': 'dgottlieb', 'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com'}Message: (cherry picked from commit 1dc2afc8b5ce09d88535049df2ad72adc0434cc8) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Gottlieb (Inactive) [ 30/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
I believe so yes.
I also believe FTDC opts out of the PBWM lock: I'm filing a follow-up ticket for FDTC to skip over sections that can't acquire locks. It's non-trivial as the AutoGetCollection constructor does not currently have a way to express lock acquisition timeouts. It also would be an API change if some sections were returned non-existent or empty. Monitoring can be prepped to handle that (if it's not already), but product would have better knowledge if omitting some data would pose a problem for customers. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 30/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'name': 'Daniel Gottlieb', 'username': 'dgottlieb', 'email': 'daniel.gottlieb@mongodb.com'}Message: | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 23/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Thanks for the clarification. Both are reported rolled together in the "Global" locks section in serverStatus and log files, correct?
I was not, and when I enable replication I do indeed see regular acquisitions reported in the Global locks metrics. Given the above I can't tell whether this is PBWM lock or actual global lock, but I assume the stack trace you quoted above shows it's indeed the global lock. Given this, then I agree the proposed change won't make a difference to FTDC.
I wonder if that's the case, or if in some way serverStatus opts out of PWBM lock. In HELP-4808 we see secondary readers regularly logging 5-10 seconds per "Global" lock acquisition, which I assume to be PWBM lock not actual global lock because a) actual global lock is rare and b) secondary is lagging indicating heavy replication activity. However FTDC is not impacted at all, suggesting it doesn't acquire PBWM lock. I'd be grateful to get clarity on this. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 23/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I believe that several replication state transitions require the global lock in mode X, but Eric's point remains. I think lockers have to opt out of participation in the PBWM lock, though. AutoGetCollectionForRead probably acquires it. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eric Milkie [ 23/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This isn't quite correct; oplog application uses a different lock than the global lock to block readers, which we call the Parallel Batch Writer Mode. We also have an easy way to allow FTDC to not be blocked by this lock. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Gottlieb (Inactive) [ 23/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Are you running with replication enabled? https://github.com/mongodb/mongo/blob/c1e7921e9d69bd9a37761deb58d119a324341a54/src/mongo/db/ftdc/ftdc_mongod.cpp#L47-L60
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 23/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
When I look at lock acquisition metrics for an idle mongod, I don't see any lock acquisitions happening at 1 per second. (There is some lock acquisition happening once every 4 seconds, but ftdc is once per second, and moreover if I increase that to 10 times per second the lock acquisitions every 4 seconds don't change, so that must be some other periodic task). So it appears that serverStatus doesn't do the normal locking as you describe and I think would be hampered by this change. For example, on a secondary ftdc would be blocked by oplog application's global lock. Can we come up with another way to accomplish the objective without negatively impacting diagnosability? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Gottlieb (Inactive) [ 23/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
My understanding is that the typical FTDC loop already does a collstats on the oplog. Unless that's special cased out of grabbing the typical locks, the proposed lock required to grab for this WiredTigerServerStatusSection is strictly easier to obtain. This proposed fix is to get the Global Lock in MODE_IS. Collstats on the oplog is probably a sequence of getting the Global Lock in IS, DBLock in IS and the collection lock in IS or S. The only time the Global Lock in MODE_IS should be difficult to obtain is for rare, tricky operations (e.g: recover to a timestamp). It's possible to add a timeout to lock acquisition and omit the wiredtiger server status section if the timeout is exceeded. Although, I expect further investigation would show the collstats on the oplog not doing such a thing. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 23/Aug/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Will this negatively impact FTDC collection by blocking it, in fact during the times when it might be particularly important to collect for diagnostic purposes? |